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Description 

[0001] This application claims priority from provisional applications Serial Nos 60/172.780. filed 20 December 1999: 
60/1 76.272, filed 1 4 January 2000: 60/1 77.432. filed 21 January 2000; 60/21 4,951 , filed 29 June 2000 and 60/21 5.000. 
filed 29 June 2000. 

[0002] This invention relates to integrated circuits, and more particularly., but not exclusively, to integrated circuits 
and methods for use with digital cameras. 

[0003] Recently. Digital Still Cameras (DSCs) have become a very popular consumer appliance appealing to a wide 
variety of users ranging from photo hobbyists, web developers, real estate agents, insurance adjusters, photo-journal- 
ists to everyday photography enthusiasts. Recent advances in large resolution CCD arrays coupled with the availabiliiy 
of low-power digital signal processors (DSPs) has led to the development of DSCs that come quite close to the reso- 
lution and quality offered by traditional film cameras. These DSCs offer several additional advantages compared to 
traditional film cameras in terms of data storage, manipulation, and transmission. The digital representation of captured 
images enables the user to easily incorporate the images into any type of electronic media and transmit them over any 
type of network. The ability to instantly view and selectively store captured images provides the flexibility to minimize 
film waste and instantly determine if the image needs to be captured again. With its digital representation the image 
can be corrected, altered, or modified after its capture. See for example. Venkataraman et al, "Next Generation Digital 
Camera Integration and Software Development Issues" in Digital Solid State Cameras: Design and Applications, 3302 
Proc. SPIE (1998). Similarly, USP 5,528,293 and USP 5 : 412,425 disclose aspects of digital still camera systems in- 
cluding storage of images on memory cards and power conservation for battery-powered cameras. 
[0004] Viewed from a first aspect, the invention provides a digital still camera architecture with features selected 
from programmable camera functions plus short audio-video clip capabilities, dual processors plus an image coproc- 
essor, burst mode compression/decompression engine, programmable preview engine, and integration of all camera 
peripherals including IrDA, USB, NTSC/PAL encoder, DACs for RGB, UART and compact flash card/smart media card 
interface. 

[0005J This has advantages including flexibility, adaptability, and efficiency. 

[0006] Embodiments in accordance with the invention will now be described, by way of example only, with reference 
to the accompanying drawings, in which: 

Figures 1a-1 b show a preferred embodiment system in functional block format. 

Figures 2-6 illustrate data flows. 

Figures 7a-7b show CFA arrangements. 

Figure 8 is a functional diagram for white balance. 

Figures 9a-9b show gamma correction. 

Figure 1 0 illustrates CFA interpolation. 

Figures 11 a- 11 b show color conversion. 

Figures 12a-12b show a memory controller data flow. 

Figure 13 is a functional block diagram of a burst compressiorV-decompression engine. 

Figure 14 is a functional block diagram of a preview engine. 

Figure 1 5 is an on screen display block diagram. 

Figure 16 is an on screen display window. 

Figure 1 7 shows a hardware cursor. 

Figure 18 illustrates a DSP subsystem. 

Figure 19 shows parallel multiply-accumulate datapath. 

Figure 20 shows a coprocessor architecture. 

Figure 21 illustrates a look-up table accelerator. 

Figure 22 is a block diagram of a variable length coder. 

Figure 23 shows a bridge. 

Figure 24 shows multiprocessor debugging support. 
Figure 25 illustrates UART connections. 
Figure 26 is a block diagram of flash card/smart card interface. 
Figures 27-38 illustrate color filter array interpolations. 
Figures 39a-39b and 40 show white balancing- 
Figures 41 a-41 b and 42a- 42e indicate image resizing. 
Rgures 43-45 illustrate tone-scaling. 
Figures 46a-46b and 47-48 show synchronization. 
Figures 49-52 show decoding buffering. 
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System overview 

[0007] Figures 1a-1b show the various high-level functional blocks in a preferred embodiment digital stil! camera 
(DSC) and systems with Figure 1b providing more detail than Figure 1 a. In particular, preferred embodiment integrated 
circuit 100 includes the following items: CCD Controller 102 interfaced with-either CCD or CMOS imager 150; preview 
engine block 1 04 to convert the data from CCD controller 1 02 into a format suitable for display using NTSC encoder 
106 or a digital LCD interface; burst mode compression-decompression engine 108 to compress the raw image data 
from CCD controller 1 02 using a lossless (or lossy, as selected by the user) compression and to write the compressed 
data to external SDRAM 160 via SDRAM controller 110. This data can then be decompressed by the decompression 
engine under DSP 122 control, processed, and displayed or stored back to SDRAM 160. DSP subsystem block 120 
(DSP 1 22 and iMX 124 plus Variable Length Coder 126 and buffers 128) performs all the processing of the image data 
in the capture mode. The data is fetched from SDRAM 160 into image buffer 128 by DSP 122 through requests to 
SDRAM controller 110. and DSP 122 performs all the image processing and compression required in the capture mode. 
The Image Extension processor (iMX) 124 acts as a dedicated accelerator to DSP 122 to increase the performance 
of DSP 122 for the imaging applications. 

[0008] RISC microprocessor subsystem {ARM 130 plus memory 132) supports the in-camera Operating Systems 
(OS). Various OSes and other real-time kernels such as VxWorks, Microitron, Nucleus, and PSOS may be supported 
on circuit 100. 

[0009] SDRAM controller block 110 acts as the main interface between SDRAM 160 and all the function blocks such 
as the processors (ARM 130 : DSP 122) : CCD controller 102, TV encoder 106 ; preview engine 104 ; etc, SDRAM 
controller 1 1 0 may support up to 80 MHz SDRAM timing and also provide a low overhead for continuous data accesses. 
It also has the ability to prioritize the access units to support the real-time data stream of CCD data in and TV display 
data out. 

[0010] Camera shot-to-shot delay is the time it takes for DSC engine 100 to read the data from CCD 150, process 
it and write it to SDRAM 1 60. The processing includes the image pipeline stages and also JPEG compression. 
[0011] In order to support real-time preview, DSC engine 100 will set CCD 150 in "fast readout" mode, process the 
data, convert the data to NTSC format, and display the data on a built-in LCD screen (not shown in Figure 1) or TV 
monitor as the case may be. 

[0012] Auto focus, auto exposure and auto white balance (the 3A functions) are performed by DSP 122 while DSC 
1 00 is in the preview mode of operation. DSP 122 reads the image data from SDRAM 1 60, performs the 3A functions 
in real-time. The algorithms for the3A functions are programmable. 

[0013] Both interlace and progressive CCD and CMOS imagers 150 interface directly to DSC engine 1O0 using the 
built-in CCD/CMOS controller 102. 

[0014] In-camera operating systems such as Microitron will be supported efficiently on ARM processor 130 in DSC 
engine 100. DSC engine 100 also has the capability to support capturing of a rapid sequence of images in the "burst 
mode" of operation. Bursts at up to 10 frames/sec of 2 Megapixel images will be supported. The duration of the burst 
sequence is only limited by the size of SDRAM 160 of the DSC system. Also, MPEG compression may be used for 
short clips. And capabilities for playback of audio-video include circular buffering. 

[0015] DSC circuit 100 also includes J/O block 140 with USB core 142 for programming and interrupt processing with 
ARM 130. 

[0016] CCD module 150 includes a CCD imager to sense the images, driver electronics and a timing generator for 
the necessary signals to clock the CCD, correlated double sampling and automatic gain control electronics. This CCD 
data is then digitized and fed into the DSC Engine 1 00. 
[0017] SDRAM 160 may be any convenient size and speed SDRAM. 

[001 8] DSC systems may be even more versatile with the ability to annotate images with text/speech. The preferred 
embodiment programmable DSP allows easy inclusion of a modem and/or a TCP/IP Interface for direct connection to 
the Internet. DSCs may run complex multi-tasking operating systems to schedule the various real-time tasks. 
[0019] Thus the preferred embodiments provide platforms for programmable camera functions, dual processors 
(ARM and DSP) plus an image coprocessor, burst mode compression/decompression engine, programmable preview 
engine, and integration of all camera peripherals including IrDA, USB ; NTSC/PAL encoder, DACs for RGB. UART and 
compact flash card/smart media card interface. Further, the platforms can provide both camera functions and digital 
audio playback on the same integrated circuit. 

[0020] The following sections provide more detail of the functions and modules. 
DSC operating modes 

[0021] The preferred embodiment systems have (1) Preview mode, (2) Capture mode, (3) Playback mode, and (4) 
Burst mode of operation as follows. 
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(1) Preview mode has data fiow as illustrated in Figure 2. ARM 130 sets CCD 150 into high-frame- rate readout 
mode (reduced vertical resolution). ARM 130 enables preview engine 104 and sets the appropriate registers for 
the default parameters. The raw CCD data is streamed into preview engine 1 04 and, after preview engine process- 
ing, is streamed into SDRAM 160. ARM 130 enables TV encoder 1 06 to display the preview engine output. Preview 
engine 1 04 processing (hardware) includes gain control, white balance, C FA interpolation, down-sampling, gamma 
correction, and RGB to YUV conversion. ARM 130 commands DSP 122 to perform auto exposure andauto white 
balance whenever required. DSP 122 processing includes auto exposure, auto white balance, and auto focus. 
ARM 130 receives new parameters for preview engine 104 and loads the preview engine hardware with these 
parameters. The output is full resolution CCIR 601 NTSC/PAL and real-time updating of gain, white balance, and 
auto focus. 

(2) Capture mode has data flow as illustrated in Figure 3a. ARM 130 sets CCD 150 in "fine" readout mode, full 
resolution. The CCD data is read directly into SDRAM 160 through SDRAM controller 110. ARM 130 commands 
DSP 122 (plus IMX 124 and VLC engine 126) perform capture processing: black clamp, fault pixel correction, 
shading compensation, white balancing, gamma correction, CFA interpolation, color space conversion, edge en- 
hancement, false color suppression, 4:2:0 down-sampling, and JPEG compression. The DSP stores compressed 
data in the SDRAM. ARM 130 writes the compressed data to compact flash/smart media 1S2. 

[0022] The computation is scheduled as two threads: iMX on one thread, the other units on the other thread. Figure 
3b shows timing and daia flow with threads related to buffers A and B. 

(3) Playback mode has data flow as illustrated in Figure 4. ARM 130 reads the compressed data from CFC/Smart- 
meda 182 into SDRAM 160 through the SDRAM controller 1 1 0 using DMA 162. ARM commands DSP 122 to do 
"playback". DSP processing (DSP 122 plus IMX 124 and VLC engine 126) includes JPEG decode (bitstream 
parsing, IDCT, VLD. and down-sampling for aspect ratio) and store uncompressed image data in SDRAM. ARM 
enables TV encoder 106 to display the image on TV/LCD display. Mote that also audio plus video (e.g., MPEG 
compressed) clips may be played back. 

(4) Burst capture mode has data fiow as illustrated in Figure 5, and Figure 6 shows offline data processing. ARM 
130 setsCCD 150 into fine resolution mode. ARM sets up the burst compression parameters, burst length, number 
of frames/second, compression ratio (lossy : lossless), etc. ARM enables burst compression engine 108 to write 
the raw CCD data to SDRAM 1 60. ARM signals DSP to process each of the stored raw CCD images in the burst. 
Burst mode decompression engine 108 decompresses each of the burst captured images. DSP processes each 
of the images as in normal capture and writes the JPEG bitstream to SDRAM 160. 

[0023] Burst capture mode is achieved by repeated calls to the regular playback routine with a different JPEG bit- 
stream each time by ARM 130. 

[0024] The preferred embodiment also has MPEG1 capture mode and playback mode. 
Image acquisition 

[0025] A DSC usually has to perform multiple processing steps before a high quality image can be stored. The first 
step is the image acquisition. The intensity distribution reflected from the scene is mapped by an optical system onto 
the imager. The preferred embodiments use CCDs, but a shift to CMOS does not alter the image processing principles. 
To provide a color image the imager (CCD or CMOS) has each pixel masked by a color filter (such as a deposited dye 
on each CCD photosite). This raw imager data is normally referred as a Color- Filtered Array (CFA). The masking 
pattern of the array of pixels in the CCD as well as the filter color primaries vary between different manufactures. In 
DSC applications; the CFA pattern that is most commonly used is an RGB Bayer pattern that consists of 2x2 cell 
elements which are tiled across the entire CCD-array. Figure 7a depicts a subset of this Bayer pattern in the matrix 
block following the CCD camera. Note that half of the pixels are sensitive to green and that the red and blue are 
balanced to green. Figure 7b shows a subset of the alternative complementary color CFA pattern with yellow, cyan, 
green, and magenta pixels. 

Image pipeline 

[0026] CFA data needs to undergo a significant amount of image processing before the image can be finally presented 
in a usable format for compression or display. All these processing stages are collectively called the "image pipeline". 
The preferred embodiment DSC may perform multiple processing steps before a high quality image can be stored. 
Most of the image pipeline processing tasks are multiply-accumulate (MAC) "intensive operations, making a DSP a 
preferred platform. The various image pipeline processing stages are described in the following sections. 
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A/D converters 

[0027] The A/D converter digitizing the CCD imager data may have a resolution of 10 to 12 bits. This allows for a 
good dynamic range in representing the input image values. Of course, higher resolution implies higher quality images 
but more computations and slower processing; and lower resolution implies the converse. The A/D converter may be 
part of the CCD moduJe. 

Black clamp 

[0028] After A/D conversion the "black" pixels do not necessarily have a 0 value due to a CCD which may still record 
some current (charge accumulation) at these pixel locations. In order to optimize the dynamic range of the pixel values 
represented by the CCD imager the pixels representing black should have a 0 value. The black clamp function adjusts 
for this by subtracting an offset from each pixel value. Note that there is only one color channel per pixel at this stage 
of the processing. 

Fault pixel interpolation 

[0029] CCD-arrays may have defective (missing) pixels, especially arrays with more than 500,000 elements. The 
missing pixel values are filled by simple interpolation. A high order interpolation may not be necessary because an 
interpolation is also performed in the CFA interpolation stage. Therefore, the main reason for this preliminary interpo- 
lation step is to make the image processing regular by eliminating missing data. 

[0030] Typically, the locations of the m issing pixels are obtained from the CCD manufacturer. The faulty pixel locations 
can also be computed by the DSC engine offline. For example : during camera initialization operation, an image with 
the lens cap closed is captured. The faulty pixels appear as "white spots" while the rest of the image is dark. The faulty 
pixel locations can then be identified with a simple threshold detector and stored in memory as a bitmap. 
[0031] During the normal operation of the DSC the image values at the faulty pixel locations are filled by a simple 
bilinear interpolation technique. 

Lens distortion compensation 

[0032] Due to non-linearities introduced by imperfections in lenses, the brightness of the image decreases from the 
center of the image to the borders of the image. The effects of these lens distortions are compensated by adjustment 
of the brightness of each pixel as a function fo its spatial location. The parameters describing the lens distortions need 
to be measured with the final system, supported by information supplied by the lens manufacturer. 
[0033] The lens adjustment can be accomplished by multiplying the pixel intensity with a constant, where the value 
of the constant varies with the pixel location. The adjustment needs to be done for both horizontal and vertical directions. 

White balance 

[0034] White balancing tries to transform the tristimulus values sensed under a certain light condition such that if 
displayed white appears again as white. In general the colors as captured by the camera do not appear on an output 
device as they were seen when capturing the scene. A couple of reasons account for that. 

[0035] First, the sensitivity of the color filters over the spectral range are slightly different. If exposed with a perfect 
white light source (constant light spectrum) the tristimulus values sensed by the CCD are slightly different. 
[0036] Second, the design of the entire CCD module and the optical system add to the imbalance of the tristimulus 
values. 

[0037] Third., typical illuminanls present while recording a scene are not constant. The illuminanLs have a certain 
"color, which is typically characterised as "color temperature" (or correlated color temperature). If an image captured 
under illuminant 1 is displayed under a different illuminant the color appearance changes. This causes a white area to 
turn a little bit red or a little bit blue. 

[0038] Several different approaches for white balancing are known. Most of them multiply the red and blue channels 
with a factor such that the resulting tristimuls value for a white patch has identical values: 
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However as explained later, this approach does not provide correction for changes of the illuminant. Therefore, the 
white balancing implementation in preferred embodiment system corrects imbalances of the sensor module. The illu- 
mination correction is handled at a later stage in the color correction section. 
[0039] Typical techniques to calculate the gain factors are 

(1) equal energy 

a1 ^ (x.,)9 2 <x,Y)'^. y) r 2 (x : y) 

(2) gray world assumption 

a1 =1 <x.y) 9(x,y)/E (Xiy) r(x,y) 

(3) maximum value in an image is white 

a1 - max (xy) g (xy) / max (Xjy) r(x t y) 

[0040] All of them do not hold in every case. Therefore, by defining the white balancing mainly as a correction of 
imager module characteristics, the algorithms to obtain the correction values can be made almost scene independent. 
[0041] The Figure 8 depicts the simplified realization of the preview engine, giving good results as long as the CCD 
sensor operates in the linear range. The white balance section below discusses a more sophisticated method. 

Gamma correction 

[0042] Display devices (TV monitors) used to display images and printers used to print images have a non-linear 
mapping between the image gray value and the actual displayed pixel intensities. Hence, in the preferred embodiment 
DSC Gamma correction stage compensates the CCD images to adjust them for eventual display/printing. 
[0043] Gamma correction is a non-linear operation. The preferred embodiments implement the corrections as table 
ook ups. The advantages of table look up are high speed and high flexibility. The look-up table data might even be 
provided by the camera manufacturer. 

[0044] With 12-bit data, a full look-up table would have 4K entries, with each entry 8 to 12 bits. For a smaller look- 
up table, a piecewise Jinear approximation to the correction curves could be used. For example, the 6 most significant 
bits could address a 64-entry look-up table whose entries are pairs of values; a base value (8 to 12 bits) and a slope 
(6 bits). Then the product of the 6 least significant bits and the slope is added to the base value to yield the final 
corrected value of 8 to 12 bits. Figure 9b illustrates a piecewise linear approximation curve, and Figure 9c the corre- 
sponding operations. 

[0045] Note that LCD displays can be considered to be linear, making gamma compensation unnecessary. However, 
LCD display modules usually expect an NTSC input (which is already gamma compensated) and hence perform some 
"gamma uncorrection" (inverse gamma correction) to compensate for this expected gamma correction. So in the pre- 
ferred embodiment DSCs using such LCD preview modules, still perform Gamma correction and then NTSC encode 
the signal before feeding it to the LCD module. 

[0046] Gamma correction may be performed at the end of the all the stages of the image pipeline processing and 
tjst before going to the display. Alternatively, the image pipeline could perform the Gamma correction earlier in the 
pipeline: before the CFA interpolation stage. 

CFA interpolation 

[0047] Due to the use of a color-filtered array (CFA), the effective resolution of each of the color planes is reduced. 
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At any given pixel location there is only one color pixel information (either of R, G. or B in the case of RGB color 
primaries). However, it is required to generate a full color resolution (R, G, and 3) at each pixel in the DSC. To be able 
to do this, the missing pixel values (R and B at the G location, etc.) are reconstructed by interpolation from the values 
in a local neighborhood in the CFA interpolation. To take advantage of the DSP in this system a Fl R-kemel is employed 
as interpolation filter. The length of the filter and the weights vary from one implementation to the other. Also the 
inlerband relationship has to be considered. Figure 1 0 describes the realization of the CFA interpolation in the hardwired 
preview engine module. It basically employs a 1 D FIR kernel for horizontal followed by vertical interpolation. 
[0048] The implementation in the DSP subsystem for high quality image processing is different in that it is fully 
programmable and able to utilize 2D filter kernels. Some background information and a proposal for an improved CFA 
interpolation technique is given in subsequent sections. 

Color correction 

[0049] Changes in the color appearance caused by differing illurninanls between capture and playback/print cannot 
be corrected just by balancing the red ; green and blue channels independently. To compensate for this, a tone (color) 
correction matrix maps the RGB pixel values to corrected RGB pixel values that take the illuminant into account. 
[0050] The principle is as follows. Let l*< denote an NxN diagonal matrix describing the recording illuminant, S the 
Nx3 matrix denoting the spectral characteristics of the imager module with one column vector for each color and R 
the 1xN column vector describing the reflectance of the scene. The measured trisLimulus value X1 at a pixel locaLion 
is given by: 

X1 T =R T *I1*S 

Denoting 
[0051] 

SS = S * S T 

we can transform the measured tristimulus value X1 into X2 : we would have been measured if the scene would have 
been illuminated by 12: 

X2 T = X1 T ' S T * SS" 1 * II' 1 * I2 * S 

The 3x3 transform matrix S T " SS' 1 * " 12 * S can be calculated offline, assuming that the spectral response of the 
sensor can be measured. Thus it is sufficient to store a set of color correction matrices for different illuminants in the 
camera. 

[0052] Since the subjective preferences of the color appearance changes among users, it is easily possible to include 
these into the color correction matrix or add a separate step to the image processing pipeline (e.g. "lone scale"). 

Color space conversion 

[0053] After the CFA interpolation and color correction, the pixels are typically in the RGB color space. Since the 
compression algorithm (JPEG) is based on the YCbCr color space, a color space transformation must be carried out. 
Also the preferred embodiment DSC generates a NTSC signal output for display on the TV and also to feed into the 
LCD preview. Hence an RGB to YCbCr color space conversion needs to be carried out. This is a linear transformation 
and each Y, Co, Cr value is a weighted sum of the R 5 G. B values at that pixel location. Figure ila illustrates the color 
conversion as realized in the hardwired preview engine. The DSP (playback) implementation is similar in principle but 
allows a higher precision conversion: 
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Edge enhancement 

[0054] After CFA interpolation the images appear a little "smooth" due to the low pass filtering effect of the interpo- 
lation filters. To sharpen the images it is sufficient to operate on the Y-component only. At each pixel focation we compute 
the edge magnitude using an edge detector, which is typically a two-dimensional FIR filter. The preferred embodiment 
uses a 3x3 Laplace-Operator. The edge magnitude is thresholded and scaled and before being added to the original 
luminance (Y) image to enhance the sharpness of the image. 

[0055] The edge enhancement is a high pass filter; this high pass filter also amplifies the noise. To avoid this amplified 
noise, a threshold mechanism is used to only enhance those portion of the image lying on an edge. The amplitude of 
tne amplified edge may vary. The threshold operation is necessary to reduce amplification of noise. Therefore, only 
those pixels get enhanced which are an element of an edge. The enhancement signal added to the luminance channel 
can be represented graphically as in Figure 11b; the parameters t1 , t2, and tne slope si can be chosen as seen nec- 
essary to obtain the best quality. 

False color suppression 

[0056] Note that the edge enhancement is oniy performed in the Y image. At edges the interpolated images of the 
color channels may not be aligned well. This causes annoying rainbow-like artifacts at sharp edges. Therefore : by 
suppressing the color components Cb and Cr at edges in the Y-component, these artifacts can be reduced. Depending 
on the output of the edge detector the color components Cb and Cr are multiplied by a factor less than 1 on a per pixel 
basis to suppress the false color artifacts. 

Image compression 

[0057] The image compression step compresses the image, typically by about 10:1 to 1 5:1 . The preferred embodi- 
ment DSC uses JPEG compression. This is a DCT-based image compression technique that gives good performance. 

Auto Exposure 

[0058] Due to the varying scene brightness, to get a good overall image quality, it is necessary to control the exposure 
of the CCD to maximize the dynamic range of the digitized image. The main task of exposure control is to keep the 
sensor operating in the linear range by controling the shutter speed } and if possible the aperture of the optical system. 
Since closing the iris and slowing down the shutter speed compensates each other, there exists a certain parameter 
range in which the exposure remains unchanged. It is obvious that this can be accomplished only to a certain extent 
as other constraints as capturing fast moving scenes may be desired by the user. 

[0059] Besides trying to keep the sensor operating in the linear range it is desirable to maximize the dynamic range 
of the ADC and hence the digitized image. This is done by controlling the PGA in the AFE. The processing necessary 
to obtain the relevant control parameters is performed on the DSP. 

Auto Focus 

[0060] It is also possible to automatically adjust the lens focus in a DSC through image processing. Similar to Auto 
Exposure, these auto focus mechanisms operate also in a feed back loop. They perform image processing to detect 
the quality of lens focus and move the lens motor fteratively till the image comes sharply into focus. Auto focus may 
rely on edge measurements from the edge enhancement previously described. 

Playback 

[0061] The preferred embodiment DSCs also provide the ability for the user to view the captured images on LCD 
screen on the camera or on an external TV monitor. Since the captured images are stored in SDRAM (or on compact 
flash memory) as JPEG bitstreams, playback mode software is also provided on the DSP. This playback mode software 
decodes the JPEG bitstream, scales the decoded image to the appropriate spatial resolution, and displays ft on the 
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LCD screen and/or the external TV monitor. 
Down-sampling 

[0062] I n the preferred embodiment DSC system the image during the playback mode after decoding the JPEG data 
is at the resolution of the CCD sensor e,g. 2 Megapixels (1600 x 1200). This image can even be larger depending on 
the resolution of the CCD sensor. However for the display purposes, this decoded data has to be down-sampled to 
NTSC resolution (720 x 480) before it can be fed into the NTSC encoder. Hence, the DSC should implement a down- 
sampling filter at the tail end of the playback mode thereby requiring additional DSP computation. 
[0063] The preferred embodiment solves this problem of additional DSP computations by a DCT-domain down- 
sampling scheme that is included as part of the JPEG decompression module. Note that the JPEG decompression 
essentially involves three stages: first an entropy decoding stage, followed by an inverse quantization stage, and finally 
an IDCT stage. In JPEG the IDCT is performed on a block of 8 x 8 pixels. The preferred embodiments down sample 
a 2 Megapixel image to NTSC resolution (a 4/8 down-sampling) in the IDCT domain by employing a 4 x 4 IDCT to the 
top left 4x4 DCT coefficients (out of a 8 x 8 DCT coefficient block) and hence effectively achieving both the IDCT and 
the 4/8 down-sampling in one step. The sampling ratio can be varied between 1/8 (smallest image) to 8/8 (full resolution 
image). 

[0064] A separable two-dimensional 4-point IDCT is applied to obtain a 4 x 4 block of image pixels from the top-left 
(low spatial frequency) 4x4 DCT coefficients. By this low-order IDCT we effectively combine anti-aliasing filtering and 
8-to-4 decimation. The employed anti-aliasing filter corresponds to a simple operation of preserving only the 16 lowest 
frequency components in the DCT domain without scaling the preserved DCT coefficients. Though this simple filter is 
effective in reducing aliasing effect, the preferred embodiments may have a lowpass filter with better frequency re- 
sponse to further reduce aliasing. The use of other lowpass filters will lead to scaling of the preserved coefficients 
where the scaling factor is the location of each DCT coefficient. 

[0065] Note that the DCT domain down-sampling technique does not increase the computational complexity. In fact, 
it reduces the computation since the JPEG decoding. stages after entropy decoding does not need to deal with the 
whole 8x8 DCT coefficients except the top-left 4x4 coefficients. Use of other anti-aliasing filters also does not add 
any complexity since the coefficient scaling operation can be merged into the low-order IDCT operation. Also note that 

this DCT domain down-sampling idea technique can offer n/8 down-sampling ratios, n = 1 7, for other CCD sensor 

resolutions. 

Up-Sampling 

[0066] Displaying cropped images for zooming of images also uses an up-sampling scheme. The inverse approach 
to the down-sampling provides an elegant tool. In the first case the 8x8 DCT coefficients are (virtually) vertically and 
horizontally extended with zeroes to form a block of NxM coefficients (N,M >8). On this block an IDCT of size NxM is 
executed yielding NxM samples in the spatial domain. 

[0067] Currently, most image pipeline operations are nonstandardized. Having a programmable DSC engine offers 
the ability to upgrade the software to conform to new standards or improve image pipeline quality. Unused performance 
can be dedicated to other tasks, such as human interface, voice annotation, audio recording/compression, modem, 
wireless communication, etc. 

[0068] Figure 27 shows a preprocessing functional block diagram including CFA interpolation, white balance, color 
correction, tone scaling,, gamma correction, conversion of RGB to YCrCb, edge enhancement, edge detection, color 
boost, and false color suppression in preparation of JPEG compression. The following sections describe preferred 
embodiments relating to CFA interpolations. 

CFA interpolation with reduced aliasing 

[0069] A preferred embodiment CFA interpolation for a Bayer pattern (Figure 7a) uses the high-frequency from the 
green channel to modify the red and blue channel interpolations to reduce the aliasing components at edges within 
the image by utilizing the signal of the other color channels. By this means artifacts are reduced, sharpness improved, 
and additional postprocessing avoided. Indeed, proceed as foflows. 

(1) apply interpolation to green channel (any interpolation method); this yields the green plane. 

(2) detect edges in the green channel (by gradient o Mother method). 

(3) compute high-pass component of the green channel (filter with any high-pass filter). 

(4) apply interpolation to the red channel (any interpolation method): this yields the red plane. 

(5) add high-pass component of (3) (with a weighting factor) to red channel. 
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(6) apply interpolation to the blue channel (any interpolation method); this yields the blue plane. 

(7) add high-pass component of (3) (with a weighting factor) to the blue channel. 

[0070] So the final image consists of three color planes: the green plane from step (1 ). the red plane from step (5), 
and the blue plane from step (7). That is : for a pixel in the final image the green intensity is taken to be the value of 
the corresponding pixel of the green plane from step (3) ( the red intensity is taken to be the value of the corresponding 
pixel of the modified red plane from step (5), and the blue intensity is taken to be the value of the corresponding pixel 
of the modified blue plane from step (7) 

[0071] Theoretical analysis of the foregoing: Each CCD pixel averages the incident optical signal over the spatial 
extent of the pixel; thus the CCD effectively provides a low-pass filtering of the incident optical signal with a cutoff 
frequency the reciprocal of the pixel size. Further the subsampling of the pixel array by the color filters on the pixels 
leads to aliasing in each color plane. Indeed, for red and blue the subsampling is by a factor of 2 in each direction; so 
the frequency spectrum folds at half the maximum frequency in each direction. Thus the red and blue baseband spectra 
areas are each one-quarter of the original array spectrum area (reflecting that the red and blue samplings are each 
one-quarter of the original array). For green ihe subsampling is only half as bad in that the spectrum folding is in the 
diagonal directions and at a distance v2 as large as for the red and blue. The green baseband spectrum is one-haif 
the area of the original array spectrum. 

[0072] Color fringing at edges is an aliasing problem, in addition, dissimilar baseband spectra lead to color fringing 
as well, even ir no aliasing is present. Indeed, aliasing is nol necessarily visible in a single color band image, but the 
effect becomes obvious upon combination of ihe three color components into one color image. The shift of the sampling 
grids between red, green, and blue causes a phase shift of the aliasing signal components. A one-dimensional example 
clarifies this: presume a one-dimensional discrete signal f(n) and two subsamplings, each by a factor of 2 but one of 
even-numbered samples and one of odd-numbered samples (so there is a shift of the sampling grids by one sample): 

f W8n (2m) = f(2m) 



f even (2m+1) = 0 



f odd (2m) = 0 
f 0 dd(2m+1)-f (2m+1) 

Of course, f(n) = f even (n) + f odd (n). Let F(z) be the z-transfarm of f (n), F eyen (z) the z-transform of f even (n) r and F^z) 
the z-transfonm of f odd (n). Then noting that F men (z) is an even function of z (only even powers of z) and F^z) an odd 
function of z (only odd powers of z): 

FevenW = ^(z) + F(-z)}/2 



F odd (z)={F(z)-F(-z)]/2 

The F(-z) corresponds to the aliasing and appears with opposite signs; that is, a phase shift of n. 
[0073] The color fringing can be reduced by a phase shift of n of the aliased components. However, this is very 
difficult to achieve, because the only available signal is the sum of the original signal with the aliasing signal. Therefore, 
the preferred embodiments have another approach. 

[0074] As long as two (or more) subsampled signals (i.e., red, green, and blue) have identical characteristics (such 
as for a gray scale image), a perfect reconstruction of the original image can be achieved by just adding the subsampled 
signals. However, in CFA interpolation generally the subsampled signals stem from different color bands. Aliasing 
errors become visible especially at edges where the interpolated signals of the different color bands are misaligned. 
Therefore, the preferred embodiments counter color fringing at edges by reducing the aliasing components only at 
edges through utilization of other ones of the subsampled signals. This reduces artifacts, improves sharpness, and 
avoids additional postprocessing. 

[0075] In particular, for Bayer pattern CFA the green channel has a higher cutoff frequency than that of the red and 
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blue channels: thus the green channel has less significant aliasing. The aliasing signal to be compensated is a high- 
pass signal, which is now estimated as the high-pass component of the green channel; and this is added (rather than 
subtracted due to the phase shift due to the offset of the red and blue subsampling grids relative to the green subsam- 
pling grid) to the red and b!ue channels. The high-pass green component could be multiplied by a scale factor prior to 
addition to the red and blue subsamplings. The signals are added while- interpolating red, blue or afterwards. 

CFA interpolation with inter-hue adaptation 

[0076] Alternative CFA interpolation preferred embodiments first interpolate Bayer pattern greens using a 5x5 FIR 
filter and then use the Interpolated green to interpolate red and blue each with two steps: first interpolate diagonally 
to form a pattern analogous to the original green pattern (this interpolation uses a normalization by the green to estimate 
high frequencies), and then apply a four-nearest neighbor interpolation (again using green normalization to estimate 
high frequencies) to complete the red or blue plane. 

[0077] More explicitly, denote Lhe CFA value for pixel location (y,x) ; where y is the row number and x the column 
number of the array, as fallows: red values R(y,x) at pixel locations (y,x) where y and x are both even integers, blue 
values B(y r x) where y and x are both odd integers, and green values g(y,x) elsewhere, that is, where y+x is an odd 
integer. 

[0078] First., let G^x) denote the green value at pixel location (y,x) resulting from the green plane interpolation; this 
is denned for all pixel localions (y,x). This interpolation can be done by various methods, including the edge preservation 
interpolation of the following section. Note that many interpolations do not change the original green values; that is, 
G A (y T x) = G(y : x) may be true for (y,x) where G was originally defined (i.e., y+x is an odd integer). 
[0079] Next, define the red and blue interpolations each ir. two steps as illustrated in Figure 28 which is labeled for 
blue and uses arrows to show interpolation contributions. 

[0080] First red step: R(y,x) is already defined for pixel locations (y,x) with y=2m f and x=2n with m and n integers; 
so first for y=2m i 1 and x=2n+1 , define R A (y,x): 

R A (y=x) = G A (y,x) {R(y-1 ,x-1) /G A (y-1 ; x-1) + R (y-1 ,x+1)/G A (y-1,x+i) + 
R(y+1 ,x-1 ) /G'Hy+l ,x- 1 ) + R(y+1 ,x+1 ) /G A (y+1 ,x+1 )] /4 

This interpolates the red plane to the pixels where B(y f x) was defined. (Figure 28 illustrates the analogous interpolation 
for blue.) Note that the this interpolation essentially averages the red values at the four corners of the 3x3 square about 
(y,x) with the values normalized at each location by the corresponding green values. If any of the green values are 
below a threshold, then omit the normalization and just average the red values. 

[0081] Perform the first blue step in parallel with the first red step because the same green values are being used. 
[0082] First blue step: B(y,x) is already defined for pixel locations (y,x) with y=2m+1 , and x=2n+1 with m and n 
integers, so first for y=2m and x=2n, define B A (y,x) : 

B A (y,x) = G A <y,x) {B(y-1 ; x-1)/G A (y-1 : x-1) + B (y-1 ,x+1 ) /G A (y-1 ,x-»-1 ) + 
B(y+1 ,x-1 ) /G A (y+1 ,x-1) + B(y+1 ,x+ tyG^y+l .x+1)}/4 

This interpolates the blue plane to the pixels where R(y,x) was defined as illustrated in the lefthand portion of Figure 
28. Again, this interpolation essentially averages the blue values at Lhe four corners or the 3x3 square about (y,x) wilh 
the values normalized at each location by the corresponding green values. 

[0083] Second red step: define R A (y.x) where y+x is an odd integer (either y=2m and x=2n+t or y=2m+1 and x=2n) 

R A (y : x) = G A (y,x) f R A (y-1 ,x) /G* (y-1 .x) + R A (y ,x+1 ) /G A (y f x+1 ) + R A (y+1 ,x) 
/G A (y+1 ,x) + R A (y,x+1) /G* (y : x+1)l /4 

This second step interpolates the red plane portion defined by the first step to the pixels where G(y,x) is defined. Again, 
this interpolation essentially averages the red values at four neighboring pixels of (y,x) with the values normalized at 
each location by the corresponding green values. 

[0084] Second blue step: define for y+x an odd integer (either y=2m and x=2n+1 or y=2m+1 and x=2n) 
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B A (y,x) = G A (y t x){B A (y-1 ,x) /G A (y-1 ; x) + B^x+I) /G A (y : x+1) + B*(y+1 : x) 
/G A (y+1 ,x) + B A (y,x+1 ) /G A (y t x+1)]/4 

5 

This second step interpolates the blue plane portion defined by the first step to the pixels where G(yx) is defined. 
Again, this interpolation essentially averages the blue values at four neighboring pixels of (y,x) with the values normal- 
ized at each location by the corresponding green values. 

[0085] The final color image is defined by the three interpolated color planes: G A (y,x), R A (y,x), and B A (y,x) . The 
10 particular interpolation used for G A (y.x) will be reflected in the normalizations for the two-step interpolations used for 
R A (y,x) and B A (y ! x). 

CFA interpolation with edge preservation 

is [0086] Alternative CFA interpolation preferred embodiments interpolate Bayer pattern greens by a (small) FIR filter 
plus preserve edges by a comparison of an interpolated pixel green value with the nearest-neighbor pixel green values 
and a replacement of the interpolated value with a neighbor value if the interpolated value is out of range. Figure 29 
illustrates the green interpolation. After this green interpolation, interpolate the red and blue planes. 
[0087] In particular firsl al each pixel (y ( x) apply the following 5x5 FIR fiilerto G(y f x) defined on the pixels (y,x) where 

^0 x+y is odd to yield G1 (y,x) defined for all (y,x): 
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The 200 center entry just implies for (y,x) where G(y ( x) is defined in the CFA, G1 (y,x) = G(y,x). Note that green values 
are in the range of 0-255, and negative values are truncated to 0. Of course, other FIR filters could be used, but this 
one is simple and effective. 

[0088] Next, for the (y ( x) where Gl(y ; x) is interpolated, consider the four nearest neighbors' values G(y±1 ,x), G(y, 
x±1 ) and discard the largest and smallest values. Let A and B be the remaining two nearest-neighbor values with B 
greater than or equal to A. Then define the final interpolated green value G A (yx) as follows: 



40 



45 



50 



55 



Ta if Gl(y,x) < A 

CMy,x> = tel(y,x) if A < Gl(y,x) £ B 

Ib if b < Gl (y,x) 

This clamps the interpolated value to midrange of the neighboring pixel values and prevents a single beyond-the-edge 
nearest-neighbor pixel from diluting the interpolated pixel value. Figure 29 shows the overall green interpolation. 
[0089] Complete the image by red and blue interpolations. The red and blue interpolations may each be a single 
step interpolation, or each be a two-step interpolation as described in the foregoing section which uses the edge- 
preserved green values, or each be some other type of interpolation. 

CFA interpolation plus noise filtering 

[0090] Preferred embodiments save on line memory required for CFA interpolation followed by lowpass filtering to 
limit noise with an integrated approach. In particular, CFA interpolation typically contains a horizontal interpolation block 
and a vertical interpolation block with line memories in between as illustrated in Figure 30. The horizontal interpolation 
block has an input of a row of CFA signals, two toggle switches, two zero insertion subblocks, two three-tap FIR filters 
(coefficients 0.5. 1 .0. 0.5). and two outputs: one output for each color. Each of the FIR filters just reproduces the input 
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color values and puts the average of successive input color values in place of the inserted 7eros. The 7ero-insertion 
and toggle timing of two subblocks alternate with each other. The block diagram of the horizontal interpolation block 
is shown in Figure 31 with a row of raw data R/G/R/G/R ... : in this block row- interpolated Red and Green signals are 
output. In case the row of raw data input is B/G/B/G/B... interpolated Blue and Green signals are output. 
5 [0091 ] Aline (row) memory delays the data by one CPA line (row) period in order to interpolate the data in the vertical 
interpolation block. Figure 32 shows the four line memories and the input/output data of the memories. In the case of 
an input row of R/G/R/G/... raw data with m indicating the (even) row number and n the column number which increments 
as the row data enters, the input and output data are: 

10 lnput_A= R(m : n) 

Output_A1 = lnput_A = R(m.n) 

Output_A2 = G(m-1 ,n) which was the interpolated green from the previous row of raw data, a G/B/G/B... row 
Output^ A3 = R(m-2,n) which was the interpolated red from the second previous row of raw data, a R/G/R/G/... row 
lnput_B= G(m,n) 
is Output_B1 = lnput_B = G(m.n) 

Output_B2 = B(m-1,n) which was the interpolated blue from the previous row of raw data, a G/B/G/B/.., row 
Output_B3 = G(m-2,n) which was the interpolated green from the second previous row of raw data, a R/G/R/G/... 
row 

20 [0092J This provides the two rows of red, R(m,n) and R(m-2,n) J for vertical interpolation to create the m-1 row of red 
and also provides the green rows G(m,n), G(m-1 ,n), and G(m-2 t n) which do not need vertical interpolation. 
[0093] The next input row (row m+1 ) of G/B/G/B/... raw data leads to the following input and output data: 

Input _A= G(mi 1,n) 
25 Output_A1 = lnput_A - G(m i- 1 ,n) 

Output_A2 - R(m,n) which was the interpolated red from the previous row of raw data., a R/G/R/G/.. row 
Output. A3 = G(m-1 ,n) which was the interpolated green from the second previous row of raw data, a G/B/G/B/... 
row 

lnput_B = B(m+1 ; n) 
30 Output_B1 = lnput_B = B(rrw-1 ,n) 

Output_B2 = G(m.n) which was the interpolated green from the previous row of raw data, a R/G/R/G/... row 
Output_B3 = B(m-1,n) which was the interpolated blue from the second previous row of raw data, a G/B/G/B/... row 

This provides the two rows of blue, B(m+1 ,n) and B(m-1 ..n), for vertical interpolation to define the m row blue and also 
35 provides the green rows G(m+1..n), Gfm.n), and G(m-1 ( n) which do not need vertical interpolation. 

[0094] Figure 33 shows the combinations for vertical interpolations. In particular, for row m output (row m+1 input) 
the combinations are (Figure 33b): 

green is G(m,n) 
40 redisR(m : n) 

blue is (B(m-1 ,n) +- B(m+1 ,n))/2 

And for row m-1 output (row m input) the combinations are (Figure 33a): 

■*5 green is G(m-1,n) 

red is (R(m,n) + R(m-2,n))/2 
blue is B(m-1 t n) 

[0095] As Figure 33 illustrates : a vertical low-pass noise filter can be applied directly to the three green outputs (G 
so (m-2,n) f G(m-1 ( n), and G(m.n) for row m input and G(m-1 ,n), G(m,n), and G(m+1 ,n) for row m+1 input), but red and 
blue cannot be vertically filtered because the four line memories of Figure 32 do not output enough lines (rows). Rather, 
eight line memories are needed as illustrated in Figure 34. 

[0096] Figures 35a-35b illustrate the preferred embodiment combination vertical interpolation and low-pass noise 
filtering including green vertical noise reduction filler block A, green-noise block B, blue/red green-noise difference 
55 block C, and red/blue green-noise sum block D. The six inputs for the preferred embodiments of Figures 35a-35b are 
the outputs of the horizontal interpolations and four line memories of Figures 30-32 and thus the same as the inputs 
to the known vertical interpolation filter of Figure 34. 

[0097] For an implementation of this interpolation plus noise filtering on a programmable processor the eight line 
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memories in Figure 34 would take up twice as much processor memory space as the four line memories of Figures 
30-32, and this can be significant memory space. For a large C FA such as a 2 megapixel (1920 by 1080 pixels) CCD, 
a line memory would be 1 -2 kbytes, so the difference would be 4-8 kbytes of processor memory. 
[0098] In more detail. Figure 35a illustrates the noise reduction and vertical interpolation for the case of input row m 
with m an even integer (raw CFA data R/G/R/G/...) into the horizontal interpolator plus four line memories of Figure 
32: the six (horizontally interpolated) inputs at the lefthand edge of Figure 35a are R(m,n), G(m-1 ,n), R(rrv2,n), G(m, 
n), B(m-1 ,n), and G(m-2.n) (i.e. , the outputs in Figure 32); and the output will be noise-reduced colors for row m-1 : R" 
(m-1 ,n) t G"(m-1 ,n), and B"(m-1 ; n). First, the vertical interpolation (lefthand portion of Figure 35a) averages R(m,n) and 
R(m-2,n) to create R(m-1 ,n); G(m-1 ,n) and B(m-1 : n) already exist as inputs. 

[0099] Then the noise reduction filter (block A in the righthand portion of Figure 35a) creates and outputs the vertically 
low-pass filtered green G"(m-1 ,n) as: 

G"(m-1 .n) ^ [G(m ; n) + 2*G(m-1 ; n) + G(m-2,n)]/4 

[0100] Next, biock B creates Deita_G as the difference between G and G"; that is, Delta_G is the vertical high- 
frequency part of G: 

Delta, G(m-1 .n) = G(m-1 .n) - G"(m-1 ,n) 

Because G is sampled twice as frequently as B and R in the Bayer CFA, direct high-frequency estimation of G will 
likely be better than that of B and R ; and thus the preferred embodiment uses Delta_G to subtract for noise reduction. 
Note that the difference between the vertical average [G(rm-1 t n) - G(m-1 ; n)] 12 and G"(m : n) equals -Dclta_G(m ( n), so 
for R and B which are to be vertically interpolated (averaged) plus low-pass filtered, the high-frequency estimate pro- 
vided by G which is to be subtracted from R and B will have opposite sign. 

[0101] Thus block C subtracts De!ta_Gfrom Bto create B" forrowm-1 because Bis not vertically interpolated for m-1 : 

B n (m-1 ,n) = B(n>1 ,n) - Delta_G(n>1 ,n) 

Essentially, the vertical high-frequency part of G is used as an estimate for the vertical high-frequency part of B, and 
no direct vertical low-pass filtering of B is applied. 

[01 02] Then block D adds Delta_G to R to create R" for row m-1 because R was vertically interpolated: 

R"(m-1 ,n) = R(m-1 ,n) + Delta J3(m-1 t n) 

Again, the vertical high-frequency part of G is used in lieu of the high-frequency part of R, and because an vertical 
averaging creates R(nv1 ,n). the opposite sign of Delta_G is used to subtract the high-frequency estimate. 
[0103] Thus the noise-reduced filtered three color output row m-1 are the foregoing G"fm-1,n), R'(m-1,n) andB"(m- 
1,n). 

[0104] Similarly for output row m from input row m+1 (again with m an even integer) and raw CFA data G/B/G/B/... 
the six (horizontally interpolated) inputs are G{m+1,n), Rfm.n), G(m-1 t n), B(m+1 t n) : G(m,n) f and B(m-1,n) ; and the 
output will be noise-reduced colors for row rn: R"(rn.n), G"(m,n), and B"(m,n). The vertical interpolation (lefthand portion 
of Figure 35b) averages B(rm-1 t n) and B(m-1ji) to create B(m f n); G(m ; n) and R(m,n) already exist as inputs. TTien 
the noise reduction filter (righthand portion of Figure 35b) block A again creates vertically low-pass filtered green G" 
(m.n) as: 

G 4, (m,n) = {G(m-M t n) + 2"G(m,n) + G(m-1,n)}/4 
Next, block B again creates the vertical high-frequency portion of G : called Delta_G, as the difference between G and G": 

Defta_G(m,n) = G(m r n) - G"(m ; n) 
[0105] Then block C again subtracts Delta_G but from R (rather than B as for row m-1 outputs) to create R": 
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R"(m ; n) = R(m.n) - Delta_G(m : n) 

Thus the high-frequency part of G is again used as an estimate for the noisy part of R, and no direct noise filtering of 
R is applied, but for row m the Delta_G is subtracted rather than added as fcr row m-1 . Indeed, for R even rows have 
Derta_G subtracted and odd rows have Delta_G added because the odd rows have R defined as a vertical average. 
[0106J Lastly., block D adds Delta_G to B to create B w : 

B"(m : n) ^ B(m : n) + Detta_G(m : n) 

Thus as with R. the Delta_G vertical high-frequency estimate is row-by-row alternately added to and subtracted from 

B instead of a direct vertical low-pass filtering of B. Note that for a given row the DeltajS terms for R and B have 

opposite signs because one of R and B will be an average of preceding and succeeding rows. 

[01 07] In short., the preferred embodiments are able to emulate the CFA horizontal interpolation , vertical interpolation. 

and low-pass filtering with only four line memories by using a high-frequency estimate based on G. 

[0108] Figures 36a-36b and 37a-37b illustrate an alternative embodiment in which the verricai low-pass filtering of 

G differs from the 1/4, 1/2 ; 1/4 weighting of the preferred embodiments of Figures 35a-35b. 

CFA interpolation for complementary color CCD 

[0109] Preferred embodiment CFA interpolations for a complementary color pattern CFA (illustrated in Figure 7b) 
combine a simple interpolation followed by an image quality enhancement by detection and adjustment for color im- 
balance. In particular, presume initial interpolation as defined at each pixel ail four complementary color values, and 
denote the color values as Ye (yellow), Cy (cyan), Mg (magneta), and G (green). 
[0110] First, at each pixel compute an imbalance factor u.: 

u. = Ye + Cy - 2*G - Mg 

This imbalance factor represents the difference between ideal and actual pixel color values. Indeed, the definitions of 
the complementary color values in terms of red value (R), green value (G), and blue value (B) are Ye = R + G, Cy = G 
+ B, and Mg = B + G. Hence, the following relation always holds for a pixel's color values: 

Ye + Cy = 2*G + Mg 

Thus the imbalance factor p ideally vanishes. When an edge is near a pixel, imbalance can arise due to the spatial 
difference of each of the four color samples in the CFA. The preferred embodiments detect the imbalance and adjust 
by modifying each color value: 

Ye' = Ye - u/4 



Cy' - Cy - u74 



Mg" = Mg + \il4 



G* = G + u./8 

Then these modified complementary colors are used to form the final image. 

[01 1 1] Figure 38 illustrates the overall flow for the enhancement using the imbalance factor Of course, scale factors 
other than -1/4, -1/4, 1/4, and 1/8 could be applied to the imbalance factor provided that Ye" + Cy ~ 2*G' + Mg'. 
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White balance 

[01 1 2] The term "white balancing" is typically used to describe algorithms, which correct the white point of the camera 
with respect to the light source under which the camera currently operates. Since the estimation of the true light spec- 
trum is very difficult, the aim of most approaches is to correct the output of the red and btue channel (assuming CCDs 
based on the RGB colorfilters). such that for a gray object the pixel intensities for all color channels are almost identical. 
The most common technique basically calculates the average energy or simply the mean for each channel. The cal- 
culation of averages may be carried out in N local windows Wj , j = 1 , 2. .... N. as for red: 

with r(k) denoting the digital signal for the red channel. Similar averages B } and Gj are calculated for the blue and green 
color channels. The imbalance between the channels : given by the green-to-red and green-to-blue ratios 

WBR= ^Gj/SjRj 
WBB= IjGj/EjBj 

are used as correction multiplier for the red end blue channels, respectively 

r'(k) = WBR r(k) 



b'(k) = WBBb(k) 

There exist many different flavors of this approach, which all calculate intensity-independent multiplication factors WBR 
and WBB. 

[01 1 3] This approach works only if several assumptions are valid. First, it is assumed that the sensor resp onses are 
well aligned over the input intensity range; in other words, the green response curve equals the red (blue) response 
curve multiplied by a factor. Looking at sensor (CCD) characteristics indicates that this assumption does not hold. For 
high fight intensities, the sensor saturates; while at very low light intensities, the sensor response (especially for the 
blue channel) is very small. Furthermore, non-linearities of the sensor, as well as imbalances of the color channels 
related to the sensor response and the light source, are handled simultaneously. Resulting artifacts include magenta 
colors in very bright areas, where the "color" should turn white, or wrong colors in dark areas. 
[01 14] The pixel intensity at the sensor output, e.g. for the red color channel, can be modeled as 

r(k) = J l(AJp(ia)f n Wa[U)dA, 

where X denotes the wavelength, the spectrum of the light source.. p(x,A.) the reflectance of the object under ob- 
servation, f R (X) the spectral sensitivity of the red color filter covering the CCD pixels, and a(U) the intensity- and 
wavelengthdependent efficiency or the CCD in converting photons into electrons. 

[0115] Regarding only the spectral response curves of the color filters 1 R {X) (and also f G (X) and f B (X)) of a typical 
CCD sensor, the output signals differ: 

WBR = J f G (yi)dA !\ f n (X)dX = 1.09 

WBB - J f G (X)d/v / f f B (X)dX = 1 .34 

The values are obtained using the response of a typical CCD and assuming perfect white light source (the spectrum 
\{X) is flat), a perfectly white object (the spectrum of the reflected light is identical to the spectrum of the illuminating 
light which means P(k ( X) = 1 ), and neglecting a(U) (no wavelength dependent quantum efficiency). Especially the blue 
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channel shows a smaller response than green or red at the same intensity. The non-linear quantum efficiency of the 
sensor is another effect. A typical s-shaped sensor response over the input intensity is shown in Figure 39a. Further- 
more, the sensor response in each channel depends on spectrum of the light source. 

[0116] Thus, preferred embodiment white balancing takes into account the misalignment as well as the non-linearity. 
Typical light sources are not flat over the visible spectrum but tend to have a higher energy in certain spectral bands. 
This effect influences the observed sensor response; ideally ft should be corrected by white point compensation, which 
may be based on a correction matrix. An independent balancing of the channels cannot handle this effect as previously 
outlined. For ease of mathematical description, approximate the s-shaped response curve in Figure 39a by piecewise 
linear segments. Three segments separate the light conditions into three categoiies: very low intensity, normal intensity., 
and very bright light. Figure 39b shows the effect of applying a single multiplier With respect to the green signal s the 
amplification of the blue signal is too small in low light conditions, whereas in very bright conditions the multiplier is too 
large. Reducing the factor leaves an offset between the components, visible as wrong colors. Therefore, the correction 
terms for aligning all three response curves must look different and reflect the sensor characteristics. 
[0117] The preferred embodiment white balancing splits into two separate schemes, one accounts for imager de- 
pendent adjustments, while the other one is related to light sources. 

[0118] Without any restrictions on generality, the s-shape response curve is approximated in the following by three 
piecewise linear segments. More segments increase-trie accuracy but do not change the basic concept. For the first 
region (very low intensity) and the blue channel, the model reads with s the response and x the input intensity: 

Sb.i = a B ^x 

Modeling the second region requires a multiplier and an offset 

^B,2 = a B,2 X + b B,2 

The offset term is determined by the constraint that the response curve needs to be contiguous at the transition point 
x., from region 1 to region 2: 

s b,i( x i) = s b.2< x i) 



sob B 2 = (a B1 -a B2 ) x-, 
The parameters for the linear model of region 3 

S B,3 = a B,3 X + b B,3 

are completely determined because the maximum output has to be identical to the maximum input x max , and the the 
response curve needs to be contiguous at the joint point x 2 : 

x max = a B,3 x max + D B.3 
S B,2( X 2) = S B.3 t X 2) 
a B t 3 = ( S B,2 ( x 2) " x max) /( x 2 ' X max) 

b B,3M1 -a B ,3) W 

Thus the parameters to specify the approximation of the response curve for each color component are a, t a 2 , x 1 . and 
x 2- x max is no1 a * ree parameter, because it is specified by the bit resolution of the input signal. 
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[01 19] The preferred embodiment white balancing now applies different multipliers for each region. For continuous 
transition from one region to the next, an additional offset is required. Although the number of regions is arbitrary, 
without loss of generality only three regions are considered in the following equations. The correction term for blue 
with respect to green for region 1 has to be: 

WBB 1 =a G1 /a B1 * Gj/B, 

where window 1 (for and B,) has pixels with intensities in region 1 . 
Thus, an input intensity value lying in region 1 gets the corrected output 

b* (k) = WBB, b(k) 

Based on the balancing multiplier for region 2 

WBB 2 = a G2 /a B 2 = G 2 /B 2 
the white balancing must consider an additional offset for values in region 2 

b* (k)=WBB2b(k) + WBOB 2 

with 

WBOB 2 = (WBBt - WBB 2 ) x 1 

For the third region the calculation is basically the same, except that no explicit WBB 3 can be specified, but the am- 
plification is determined by the maximum value x max . 

b* (k) = WBB 3 b(k) + WBOB 3 

with 

WB B 3 - (* max - (WBB 2 x 2 + WBOB 2 ))/{x max - x 2 ) 
WBOB 3 ^(1-a B3 )x max 

For an implementation, thesystem must determine appropriate white balancing multipliers WBBj for N-1 regions. Based 
on these values, the remaining offset values WBOB and the multiplier for the lasl regions are calculated. The locations 
of the transition points are specified a priori. The white balancing itself selects the region based on the intensity value 
of the input pixel and applies the appropriate gain and offset to that value 

f WBBi*b(k) b(k) < Xi 

b' <k) = i WBB 2 *b(k) + WB03 2 x t < b(k) < x 2 
I WBB 3 *b(k> * WBOB3 x 2 < b(k) 

Plus a similar multiplier for the red channel. 

[0120] The total dynamic range of the CCD output signal is independent of aperture, and shutter, since they affect 
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the number of photons captured in the CCD. An analog gain however, or any digital gain prior to processing shifts the 
signal and should be avoided. In case a gain (digital) a needs to be applied, this gain can be included Into the white 
balancing method. A gain maps ihe maximum input value x max to the output value a'x max 

[0121 ] The scaled response curves behave identical to the non-scaled one. meaning that the scaled signal saturates 
at a x max . Substituting 

WBB 1 ; = a*WBB 1 



WBB 2 : ^ a*WBB 2 
In thai way the equation in the previous section remain unchanged, except 

WBOB 3 = (a-a Bi3 )x max 

After linearization Lhe signal can undergo an adjustment reflecting the light source. This is also known as white poinl 
adjustment. Here the input signal is transformed such that it (ooks like as if it has been captured under a different light 
source. For example, an image has been captured in bright sunlight (D65), but the color characteristics should be as 
if it has been captured under indoor conditions (D 50 tungsten). 

[R,G,B] D 6 / = ! D65 T *p*[f R: f G : f B ] T 

[R.G.B] D 50 T =l D50 T -p*|T Rl f Gl f B ] T 

Here. I DXX denotes a vector sampling the light spectrum, p is a diagonal matrix describing the reflectance of th e objects, 
and f R , f G , and f B denote the spectral response of the CCD light filters. Based on these equations a 3x3 transformation 
matrix can be calculated relating the signal under D65 to D50: 

[R, G, B] D 50 T = l D50 T * l D65 " T * [R : G, B] D 65 T 

The 3x3 transformation matrix 

m d = 'dso 7 * ! D65 1 

can be calculated offline. 

In real systems it is almost impossible to determine averages for the different response regions. Therefore a simple 
solution is to calculate overall values as in the foregoing ratio of integrals, and modify them with fixed values based on 
predetermined sensor measurements 

WBB 1 = a, * WBB 



WBB 2 = a 2 * WBB 

And similarly for WBR. 

The transition points can be fixed in advance, too. There is just one exception for the transition pointx 2 . In rare situations 
the WBR-value may be so large that it exceeds the maximum output value at the transition point x 2 . In that situation, 
either the WBR needs to be decreased or the transition point is reduced. The diagram in Figure 40 shows an example 
of the effectiveness of this technique. The red components is adjusted with respect to the green component Using a 
single multiplier exceeds the green signal "m bright areas, and is less effective in low light areas, whereas the segmented 
white balancing matches the green curve for all intensities. 
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Resi7ing preferred embodiments 

? 0122] Frequently images captured in one size (e.g.. 320 x 240 pixels) have to be converted to another size (e.g.. 
-bout 2B8 x 21 6) to match various storage or input/output formats. In general this requires a fractional up-sampling or 
down-sampling by a rational factor N/M; for example ; a resizing from 320 x 240 to 288 x 21 6 would be a 9/10 resizing. 
Theoretically, resizing amounts to cascaded interpolation by N, anti-aliasing filter, and decimation by M. In practice the 
resizing may be achieved with an M-phase,. K-tap filtering plus selection of N outputs per M inputs, 
[0123] For example, preliminarily consider a resizing by a ratio of 63/64 using a 3-tap filter as illustrated in Figure 
41a in which the top horizontal line represents pixel inputs and the horizontal length-three braces represent the 3-tap 
filter kernel applied to the indicated three inputs and producing the indicated outputs. Indeed, presume the filter kernel 
is a continuous function f (t) with support of length 3-1/63 so that at most three inputs can be involved; see Figure 41 b. 
Note the slight shifting to the right of successive braces in Figure 41a: this represents the resizing from 64 inputs down 
to 63 outputs because the center of the filter kernel (and thus the non-rounded-off output position) must increment 
1 +1 /63 (=64/63) pixel positions for each output in order forthe 63 outputs to match the 64 inputs. Output[0] (represented 
by the farthest left brace in Figure 41a) is centered at the position of input, and the non-rounded-off output position j, 
denoted outp_pos|jJ, thus equals 1 + f 64/63, 

[0124] The filter kernel is represented as a symmetrical continuous function f(t) centered at time 0. Output[0] for 
example, needs three kernel values: f(-1), f(0), and f(1). Each output point is computed as the inner product of three 
kernel coefficient values with three input pixel values. The center input point for the oulput[j] is positioned at round 
(outp_pos[j]) where roundQ is the round off function. The other two input points are offset from this center point by ±1 . 
The center filter kernel coefficient value is f(round(outp_pos[j])- outp_pos [j]) and the other are f() at the +1 offsets of 
this center value point. Thus the following table shows the output position, coefficient kernel values, and input points 
needed for each output: 



output j 


outp_pos 


center coeff position 


input points 


0 


1 


0 


0,1,2 


1 


2 1/63 


-1/63 


1,2,3 


2 


3 2/63 


-2/63 


2 : 3,4 










31 


32 31/63 


-31/63 


31 ,32,33 


32 


33 32/63 


31/63 


33,34,35 


33 


34 33/63 


30/63 


34,35,36 










61 


62 61/63 


2/63 


62 : 63 f 64 


62 


63 62/63 


1/63 


63,64,65 


63 


65 


0 


64,65,66 











The table shows the desired coefficient position as well as 1he inputs involved in each output. Note the j=63 case is 
similar to the j=0 case in that the kernel center aligns with the input, but with the output position and input indices shifted 
by 64. Notice that at j=32 there is a change in the input pattern: for j<31 , outputQ] uses input j, j+1 , and j+2; whereas 
for j>32, output[j] uses inputs j+1, j+2, and j+3, 

[0125] The preferred embodiments partition the filtering computations for resizing a two-dimensional array (image) 
between iMX 124 and DSP 122 and limit memory use as follows. First iMX 124 performs the 3-tap row filtering with 
64 banks of coefficients and then 3-tap column filtering with 64 banks of coefficients. First consider the row filtering. 
3-tap row filtering on iMX 124 has the input/output relationship: 



iMX output] 


input points 


0 


0,1.2 


1 


1,2,3 
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(continued) 



iMX output j 


input points 


2 


2,3.4 






31 


31,32,33 


32 


32,33.34 


33 


33 r 34.35 






61 


61,62.63 


62 


62.63.64 


63 


63 ; 64.65 


64 


64 ; 65 : 66 







Comparing this table with the prior 63/64 resizing table shows that the only difference is Lhe iMX produces one extra 
point, namely, IPP_output[32]. Thus the preferred embodiments produce the 64 output points with iMX 124 ; and then 
use DSP 122 to pick the 63 valid points: 
output Q] = JPP^outputQ] 

forj * 1 ( 2,...31 IPP_outpt[j+1] forj ^ 32 ; 33 62 

In general, N/M resizing when N/M is less than 1 involves deleting M-N outputs of every M outputs. Thus the preferred 
embodiments generally perform the filter operations on the M input points in an accelerator such as the iMX and then 
use a processor such as the DSP to discard the unneeded outputs. (iMX can also handle targer-than-unity resizing up 
to N/M = 3.) 

[0126] iMX can produce 8 outputs of 3-tap row filter in 3 cycles. Basically, 8 adjacent outputs are computed in parallel 
using the 8 MAC units. At time 0, pull out input points 0,1 ,2,3,. ..7, multiply with appropriate coefficients (each can be 
different), and accumulate into 8 accumulators. At time 1 pull out input points 1,2,. ..8, do the same, and at time 2, pufl 
out input points 2,3.. ..9. accumulate the products, and write out 8 outputs, j=0.1 ,...7. Next, shift over 8 input points to 
compute j=8,9,...15. 

[0127] For the vertical direction, iMX computes 8 outputs in parallel as well. These are 8 horizontally adjacent output 
points, and every fetch of input array also bundles 8 horizontally adjacent output points. Therefore, all 8 MAC units 
share the same coefficient values for each cycle. For vertical direction there is less data reuse in iMX, so input/output 
memory conflicts slow down the computation to 4 cycles/8 outputs. Total filtering time is 7 cycles/8 outputs, or 7/8 cycle 
per output. 

Input data is of size 320x240x3. Thus, the filtering of iMX takes 320x240x3.7/8 201 ,600 cycles, or 1 .7 msec with iMX 
running at 120 MHz. 

[0128] Afterfiltering : DSP picks correct outputs. Basically, one row out of every 64 rows and one column out of every 
64 columns should be discarded. A DSP assembly loop moves the valid iMX output points to a separate output area. 
iMX and DSP may run in parallel if there is sufficient local memory for both. An entire input image likely is too large to 
fit into local memory; even the natural choice, 63x63 output points, may be too large. In such a case partition the image, 
such as 63 wide x 16 tall, and deal with extra bookkeeping in the vertical direction. With just 3x64 = 1 92 coefficients, 
it would be economical to pre-compute and store them. DSP should keep track of the phase of each processing block, 
and point iMX to the correct starting address of coefficients. If the colors are interleaved, this allows interleaved filtering 
as well. iMX deals with strides in-getting input points. The following table shows interleaved 3-tap filtering. 



J 


input points 


0 


0,3,6 


1 


1.4,7 


2 


2,5,8 
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However, interleaving consumes three times more memory for the same output block si7e for each color. Thus it si 
possible to partition the task into smaller size, such as 63x5 on each color plane, and eal with extra overhead in the 
vertical direction. If the color format is not 4:4:4 (say, 4:2:2). and input is colorinterieaved. the DSP will need to spend 
some additional time separating color planes. 

[0129] Performing resizing totally in DSP 122 is time-consuming if implemented with straightforward fractional ad- 
dressing. The preferred embodiments streamline the computation by requiring filter coefficients to be reordered and 
padded with dummy words, iMX 124 performs the main processing concurrently with DSP 122 computing the coeffi- 
cients. This efficiently realizes high throughput resizing. 

[0130] In more detaiL the preferred embodiments perform an N/M resizing of an image by using iMX 124 to perform 
M-phase. K-tap filtering (which produces redundant output points) and DSP 122 to select the correct output points. 
Further, DSP 122 computes needed coefficients from a fewer-subsample coefficient template to reduce memory usage 
to 8*K; otherwise memory usage up to 2*lvTK coefficient words would be needed. DSP 122 can compute the rounded 
position for the coefficients, and build up the coefficient memory for iMX 124. 

[01 31 ] For processing wide and shortblocks of pixels (i.e., 1 6 x 64) the horizontal direction requires more computation 
in that horizontal coefficients are updated more often than vertical coefficients. However the coefficients constructed 
by DSP 122 can be reused many times within the short block, so the load on DSP 122 should not be excessive. 
[0132] In particular preferred embodiments proceed with the following steps which are illustrated In Figures 42a- 
42e tor a 3tap filter and a 10-to-9 resizing (e.g., resizing from 320x240 to 288x216 in 30 frames/sec) (presume 4:4:4 
inlerleayed, for 4:2:2 or 4:1 ;1 do subsampling after resizing): 

1 . select input/output pattern: every 10 inputs leads to 9 outputs as per Figure 42a. 

2. draw coefficient pattern for a processing unit, one color first. Arrows in Figure 42b indicate which input points 
are used: connected arrows form the same output point, and gray (open head) arrows indicate zero coefficients. 
Thus three input points determine the first output point only two input points determine each of the next eight 
output points, and then a tenth ignored output (no non-zero input points); and this repeats every ten. This pattern 
suggests use of a polyphase 3-tap filter, and drop the last output in every group of 10 outputs. 

3. consider interleaved input/output. See Figure 42c which shows a set of three groups of ten input points inter- 
leaved so that the three input points determining the first output point from the original first group of ten input points 
are now at locations 1 , 4, and 7; the three input points determining the first output point from the original second 
group often Input points are now at locations 2, 5, and 8; and the three input points determining the first output 
point from the original third group of ten input points are now at locations 3. 6. and 9: and so forth. This interleave 
implies that sets of three adjacent output points use all different input points and do not require simultaneous 
memory accesses. 

4. Consider 8-way parallelism and iMX, add more dummy outputs if necessary. See Figure 42d which shows the- 
output points partitioned into four groups of 6 for parallel computations, 

5. Compute coefficients and order as grouped. iMX will process one group at a time, using coefficient order from 
left-toright then up-to-down, then next group. Coefficients need to be arranged to the same order. If the iMX 
coefficient memory and the flash memory can accommodate all these coefficients, these coefficients can be in- 
cluded in the DSP code as constant data, and this step is done once in the software development. If the iMX 
coefficient memory can hold these coefficients all the time, but these take up too much room in the flash memory, 
this step can be performed once during system initialization. Likely the SDRAM can hold all these coefficients., but 
iMX coefficient memory cannot hold them all the time, this step should be performed Dnce in the system initialization, 
an the coefficient image should be stored in SDRAM. When needed, these coefficients are swapped in from the 
SDRAM. If it is not desirable to store all these coefficients at any time, especially when M is very large (100+), 
compute needed "window" of coefficients with DSP concurrently with iMX processing. Just make sure the iMX 
coefficient memory can hold the necessary coefficients for a computation block. 

6. Start computation on iMX. In this case, it takes about 12 cycles in the inner loop to produce the 27 valid output 
points. Each iMX command can produce a 2-D output block, so producing 16x27 output points will take about 10 
-»- 16*12=202 cycles. 

7. When iMX is done, have DSP pick the correct output points. In this example, 276 points are picked out of every 
group of 32 output points. This task will be easier to code if the width of output matches or is a multiple of 3'M. 
DSP only has to touch each valid outaput once : so the loading of the DSP should not be significant. 

[01 33] In vertical resizing, iMX works in SIMD mode. Every group of 8 adjacent data input are processed in parallel. 
Coefficient are used one value per cycle, and this value should apply to all color components. Even if resizing factors 
are the same for horizontal and vertical, how iMX uses coefficients is different, so there needs to be a separate vertical 
resizing coefficient storage (which takes 1/3 of horizontal coefficients). See Figure 42e. Again, there is the option to 
keep all vertical coefficients in iMX, swap in and out, or have DSP compute on the fly, DSP may need to pick valid 
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output rows after iMX completes processing. 
Tone-scaling preferred embodiments 

[0134] Tone-scaling operates on the dynamic range of the luminance signal (or the color signals) of an image lo 
make details more clear. For example, a picture taken against the light or in a very bright environment typically has 
high brightness levels. Tone-scaling commonly relies on luminance (or color) histogram equalization as illustrated in 
block form by Figure 43. Indeed, converter block 430 converts the input luminance levels (in the range 0 to 255 for 
8-bit or 0 to 4095 for 12-bit) to output luminance levels in the same range using a look-up table. The look-up table 
consists of the pairs that are the input level and the corresponding output level with the output levels calculated in 
histogram equalization block 432 as follows. First, find the cumulative distribution function of the input luminance levels 
of the image to which the tone-scaling will apply; that is, find F(r) such that F(r) = (the number of pixels with level < r) 
/(total number of pixels in the image). Next, create the look-up table function T(r) through multiplication of F(r) by the 
maximum pixel level and round-off lo the nearest integer. Then the look-up table is just the pairs of levels (r,s) where 
s = T(r). Figure 45 illustrates T(r) for an under-developed image (the majority of pixels have a low level as reflected by 
the large slope of T(r) for small r) in which fine details in dark parts are difficult to perceive. Also as Figure 45 shows 
for this under-developed image, the tone-scaling converts the level r = 500 to s = 2000; and thus in the tone-scaled 
image the differences of the luminance levels will be emphasized for the low levels and de-emphasized for the high 
levels. Thus the tone-scaling enhances detail in dark portions. 

[0135] However, the tone-scaled image may look unnatural in that the colors are too clear, as if the tone-scaled 
image were painted in oil paints. Thus this lone-scaling is sometimes too strong for consumer use because of the 
unnatural character even if the fine details are clearer; although other applications such as medical and night vision 
demand the fine detail despite unnaturalness. 

[01 36] The preferred embodiments provide tone-scaling by using a linear combination of the histogram equalization 
function T(r) and the original image level r. That is, for a parameter a with 0 < a < 1 define a tone-scaling function by 

s= Round( aT(r) + (1-a)r) 

where T(r) is as previously described except that the round off to the nearest integer is not needed in the definition of 
T(r) because of the subsequent multiplication by a plus addition of (1 a) r and round off. Figure 45 illustrates the preferred 
embodiment for a = 0.3 between the curve s = T(r) and the identity line s = r. 

[01 37] Figure 44 shows preferred embodiment tone-scaling in functional block form: again define a histogram equal- 
ization function T() for the luminance (or color) levels in block 442 : and then define the rounded-off linear combination 
with weight a of T() and the identity in block 444 to yield the final look-up table for the tone-scaling in converter 440. 
When the weight a equals 0, then there is no tone-scaling and a natural iook, but when the weight a equals 1, the 
tone-scaling is with T() and fine details are enhanced. The value of weight a can be selected according to the application. 
All of the computations are programmable. 

Implementation details 

[0138] Preferred embodiment hardware structures supporting the foregoing functions include the following. 
SDRAM Controller 

[0139] SDRAM controller block 110 acts as the main interface between SDRAM 1 SO and all the function blocks such 
as processors (ARM 130, DSP 122), CCD controller 102, TV encoder 106, preview engine 104, etc. It supports up lo 
80 MHz SDRAM timing. It also provides low overhead for continuous data accesses. It also has the ability to prioritize 
the access units to support the real-time data stream of CCD data in and TV display data out. It also provides power 
down control for external SDRAM . DSP 122 can inhibit CKE signal of SDRAM 160 during no data access. 
[0140] SDRAM controller block 110 supports 16/54/128/256 MB SDRAMs, 32-bit width or2 x 16-bit width SDRAMs, 
maximum BO MHz (e.g., 1 0-80 MHz) operation, availability of word, half-word, or byte access (ARM), commands: mode 
setting, power down and self refresh, programmable refresh interval, 2 or 3 CAS latency can be selectable, 2 Chip 
Select Output (maximum SDRAM size is 1G bit), authorizes and manages DMA transfers, manages the data flow 
between processors SDRAM, CCD data buffer to SDRAM, preview engine to SDRAM, burst compression to/from 
SDRAM, video encoder from SDRAM, OSD from SDRAM, ARM to/from SDRAM, DSP image buffer to/from SDRAM. 
Figure 12a shows the data flow managed by the SDFtAM controller. The signals and priorities are: 
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i Signal Name 


Signal Description 


cik 


SDRAM clock (10-80 MHz) 


Req 


Data read/write request signal 


req_en 


Request enable (acknowledge) signal from SDRAM Controller 

When the peripheral modules require a data IN/OUT, the req signal shall be asserted and when the 
req_en signal is asserted, the req signal shall be negated 


Address 


Start address of read or write CCDC, PREVIEW, BURSTC, ENC : OSD, DSP: 22-bit width 
ARM: 25-bit width 


Odata 


output data to SDRAM (32-bit) 


I data 


Input data from SDRAM (32-bit) 


Rw 


Read or Write signal 
0: Write / 1: Read 


Dten 


Data write enable signal for DSP IF 


Ds 


Bus Select (4-bit) for ARM IF 



The Priority list of access units is as follows, 



Priority 


Access Unit 


1 (highest) 


ENC out 


2 


CCD in 


3 


OSD out 


4 


PRVWin 


5 


BURST in 


6 


DSP I/O 


7 


ARM I/O 



Preview engine 

[0141] Figure 14 is a block diagram of preferred embodiment preview engine 104 which provides image data with 
YCbCr in 4:2:2 format from CCD raw data from CCD-controller 102 and has the following main functions. 

Available for both RGB CCDs and complementary (YeCyMgG) CCDs (Figures 7a- 7b show these CCD patterns) 
Digital gain adjustment 
White balance 

Vertical and horizontal noise filter 
RGB gain adjustment for complementary CCDs 
Independent gamma correction for RGB colors 
- YCbCr-4:2:2 formatted data output 

[0142] Sync module 1 402 generates control signals for other modules such as a sync signal for a starling point of 
an image and an enable signal for down sampling. In this module, no image processing is executed. White balance 
module 1404 executes digital gain adjustment and white balance for CCD raw data. CFA interpolation module 1406 
has many important sub-modules such as a horizontal noise filter, a horizontal interpolation, a vertical noise filter, a 
vertical interpolation, a down sampling, etc. This module outputs RGB formatted data irrespective of CCD mode (RGB 
CCD or complementary CCD). RGB gain modules 140B for complementary CCD allow adjustment to white balance 
by RGB color format for complementary CCD. Gamma correction modules 1410 execute gamma correction with an 
approximated gamma curve having 4 linear segments. This module exists for each color to permit the independent 
adjustment to RGB. RGB2YCbCr conversion module 1412 converts RGB formatted data into YCbCr formatted data 
and adjusts offsets of Cb and Cr. 4:2:2 conversion module 141 4 converts YCbCr-4:4:4 formatted data into 4:2:2 format 
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and outputs them on a 32-bit data bus. SDRAM interface module 1416 communicates with SDRAM controller 110 
(Figure 1b) and requests it to store YCbCr-4:2:2 formatted image data. 

The following describes the modules. 

[0143] White balance module 1404 executes digital gain adjustment and white balance for CCD raw data. Digital 
gain adjusts for total brightness of the image and white balance adjusts the ratio of colors existing in a CFA pattern. 
[0144] Figure 8 is a block diagram of white balance module 1404. There are two multipliers for the two gain adjust- 
ments and clip circuits to reduce the size of circuits. A gain value lor digital gain named PVGA1N in this figure uses 
data in a PVGAIN register and white balance is selected automatically by setting the CFA pattern register. 
[0145] CFA interpolation module 1406 include both sub-modules for horizontal and vertical interpolation and for 
horizontal and vertical noise filtering, down sampling, color adjustment and complementary color to RGB color con- 
version. Figure 10a is a block diagram of CFA interpolation module 1406. Horizontal noise filter sub-module 1002 
executes a three-tap low pass filter horizontal filter; see Figure 10b. 

[0146) Horizontal interpolation filter sub-module 1004prepares two types of filters and interpolates horizontally using 
one of them. The ouiputs signal "L" and "R" means a left data and a right data on the line. For example, a processed 
line starts the following CFA pattern, GBGBGBGBGB .... the output signal "L" is G and "R" is B. Therefore, these two 
outputs change the colors each line. Horizontai down-sampling sub-module 1006 outputs only data on valid pixels 
based on register selLings of horizontal decimation pattern. Vertical interpolation sub-module 1008 processes a Lhree- 
tap vertical interpolation filter using two line-memories 1010 outside the preview engine module and outputs data of 
all colors existing in the CFA pattern. And this sub-module also executes a vertical noise filter. Color selection sub- 
module 1 01 2 extracts data by each color in the CFA pattern and outputs RGB color formatted data in RGB CCD mode 
or complementary color formatted data in complementary CCD mode. In this figure, "g" signal is temporal data regarding 
G and used for recalculating R and B in the next color adjustment sub-module 1014. The color formatted data is 
processed color adjustment in color adjustment sub-module 1014 and the processing is different depending on CCD 
mode. This image processing from vertical interpolation sub-module 1008 to color adjustment sub-module 1014 has 
a strong correlation depending on CCD mode and vertical interpolation mode. Therefore, the processing should be 
considered as a sequence of vertical interpolation processing as described below. Comp2RGB conversion sub-module 
1016 converts complementary color format into RGB color format in complementary CCD mode. In RGB CCD mode, 
the data bypass this sub-module. 

[0147] The following sections describe these sub-modules. 

[01 48] Horizontal noise filter 1 002 executes three-tap horizontal low pass filter and can reduce random noise effec- 
tively. Actually, when the center of data is set to X 0l the following calculation is executed depending on the CFA pattern 
and its processed line. 

f (X-2 + 2X 0 + X 2 )/4 (two colors in processed 

line) 

l(X_i + 2Xo + (one color in processed line) 

An on/off switching of this filter can be controlled by a register setting. 

[0149] Figure 10b is a block diagram of horizontal noise filter sub-module 1002. The two types of filter are imple- 
mented by using two adders and a switch named °three_taps_sw" in this figure. If there is one color in the processed 
line, the switch is set to on (High in the figure). This switch is automatically controlled depending on a register setting 
of the CFA pattern and a position of the line in the processed image. Before the output, noise-filtered data or bypassed 
data is selected by a register setting. 

[0150] In horizontal interpolation sub-module 1004, there are two modes of filtering and the data from horizontal 
noise fitter 1002 is interpolated horizontally by either a two-tap or five-tap interpolation filter. The two-tap fitter utilizes 
the average the two data at the adjacent pixels on the left and right to interpolate the center data. This mode is called 
"simple mode". The five-tap horizontal interpolation filter utilizes the information of another color on the processed line 
so that a false color around an edge in processed image can be reduced effectively. This mode is called "normal mode". 
These modes are selectable by a register setting. Actually, when the center of data is set to X 0 , the following calculation 
is executed depending upon the interpolation mode. 



25 



EP 1 111 904 A2 

f (-X-2 + 2X. 1 + 2X 0 + 2Xi - X 2 )/4 



[normal mode) 



x 0 = i 

l(X.i + Xi)/2 (simple mode) 

Figure 10c shows an exampie of this horizontal interpolation processing in RGB Bayer CCD mode. In this figure, 
interpolated data is represented by small letters. 

[01511 Figure 10d is a block diagram of horizontal interpolation module 1004. Two adders, one subtracter and a filter 
mode switch are implemented for executing one of these two types of filters. The filter mode switch is controlled by 
setting a register. 

[0152] Vertical interpolation sub-module 1 00B processes either a two-tap or three-tap vertical interpolation fitter using 
two line-memories outside the preview engine module and outputs the Information of all colors existing in the CFA 
pattern. And this sub-module also executes a vertical noise filter. An image processing in this module is a little com- 
plicated and the outputs from this sub-module is varied depending on a processed line, CCD mode, CFA pattern, filler 
mode and noise filter on/off. As explained in the following, the image processing from vertical interpolation sub-module 
1008 to color adjustment sub-module 1014 has a strong correlation and this processing flow of them should be con- 
sidered as a sequence of vertical interpolation processing. Therefore., this sequence of the vertical interpolation 
processing is explained first. The sequence may be called "vertical interpolation sequence". 

[0153] As with horizontal interpolation, vertical interpolation processing also has two types of interpolation mode, 
that is "simple mode" and "normal mode". An interpolation filter in simple mode utilizes the average two data at the 
next pixels on the upper and lower to interpolate the center of data. In normal mode, the processing differs between 
RGB CCD mode and complementary CCD mode. The interpolation filter in normal mode in RGB CCD mode utilizes 
the data of one of the others color same as horizontal interpolation filter Actually, when the data of a certain color to 
be interpolated is set to X (mainly R,B) and the data of a color utilized as a reference is set to Y (mainly G), the following 
calculation is executed depending on the interpolation mode through this vertical interpolation sequence and it is the 
output from color adjustment sub-module. 

f (X-i - Y-! + Xj. - Y L )/2 + Y 0 (normal mode) 

xo ={ 

I (X-i + Xi) 12 (simple mode) 

Figure 10e shows an example of this vertical interpolation sequence for the RGB Bayer CCD pattern. 
[0154] In complementary CCD mode, normal mode means "simple interpolation with color adjustment". That is, data 
of all colors which is processed by simple vertical interpolation is adjusted based on the formula in complementary 
color space Actually, when the data of a certain color to be interpolated is set to X and the data of the others color is 
set to W ( Y, and Z the following calculations are executed in normal mode in complementary CCD mode. 

f (X_i + Xx)/2 ± a(w 0 , x 0 , y 0 , Zo) (normal mode) 
I (X-i + Xi)/2 (simple mode) 

As to the calculation of a = a(w 0 , x 0 , y 0 , z 0 X see below. 

[0155] In this vertical interpolation sequence, main roles of vertical interpolation sub-module 1008 are to execute a 
part of vertical interpolation sequence and vertical noise filter. The part of vertical interpolation sequence means pre- 
paring data for normaJ vertical interpolation mode. As shown in Figures 10e and 10b (for RGB and complementary 
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CCD patterns, respectively), in simple mode an output data of this vertical interpolation sub-module bypasses color 
adjustment sub-module. Therefore, in simple mode, the output from this sub-module is used as the output of vertical 
interpolation sequence. In any case of interpolation mode, this sub-module calculates the following equation for vertical 
interpolation sequence. 

Xo = (X.i 

Vertical noise filter ... which executes the following 3 taps vertical low pass filter is also processed in this sub-module 
depending on the CFA pattern, 

XqMX.t -2X 0 -X 1 )4 

However, for this filtering, data of same color on processed 3 lines must be prepared. Therefore, a function of the 
vertical noise filter mainly executes only G in RGB Bayer CCD. Figure 1 0g shows an example of the output of this 
vertical interpolation sub-module for a RGB Bayer CCD. When the vertical noise filter can be applied and it is set on, 
original data (R in this figure) Is also adjusted in order to keep a correlation to the others color (G in this figure). 
[0156] Figure 10h is a block diagram of vertical interpolation sub-module 1008. Six adders and two subtracters are 
implemented for executing vertical interpolation and noise filtering. Especially, a calculation process of L_121 and 
R_121 is so complicated that switching operation for 1__121 and R_121 -is not shown to simplify this figure. 
[0157J Color selection sub-module 1 012 arranges the inputs from vertical interpolation sub-modules in order of color 
format, that is R, G and B in RGB CCD mode or Yg, Cy, Mg, G in complementary CCD mode. This arrangement is 
executed automatically by setting register of the CFA pattern. Figure 10i shows an example of this color selection 
processing in RGB Bayer CCD of Figure 10g. The outputs named V in this figure is a temporal data of G and is used 
for recalculation of R or B in RGB CCD mode in color adjustment sub-module. 

[0158] Figure 10j is a block diagram of color selection sub-module 1012. Four color extractors switch and select 
independently correct colors from four inputs from vertical interpolation sub-module 1 008. 

[0159] Color adjustment sub-module 1014 executes the rest of calculation for vertical interpolation sequence. In 
RGB CCD mode such as RGB Bayer CCD, R or B is recalculated using the temporal data of G. When data of R or B 
from color selection sub-module is set to X, the following calculation is executed in RGB CCD mode. 

In the example of Figure 1 0i, when noise filter is off, 

x =( b o 2 - b 22>' 2 



G temp = ( G 02 + G 22> 12 
G = 912 

Therefore, 

x = B 

= ( b 02 * D 22) 12 - < G 02 + G 22) /2 + 9l2 
= (( b 02- G 02) + ( b 22- G 22)) /2 + 9l2 

This is the output B of the color adjustment module and also the output of vertical interpolation sequence. That is, 
vertical interpolation sequence in RGB CCD mode utilizes the average of differences between data of color to be 
interpolated and reference data of the others color. 
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[0160] In complementary CCD mode, color adjustment is processed to data of all colors from color selection sub- 
module. First, value a is calculated at each pixel based on a formula in complementary color space Ye + Cy = G + Mg. 

a =G+Mg-Ye-Cy 

That is, the value a can be considered as the amount of an error value of four colors. Therefore, in complementary 
CCD mode, to data of all colors, Ye, Cy, Mg and G, the following adjustment is processed to satisfy the above formula. 

ye = Ye + a'4 



cy - Cy + a'4 



g = G - a/4 



mg = Mg - a/4 

Figure 1 0k is a block diagram of color adjustment sub-module 1 01 4. Six adders and three subtracters are implemented 
for executing the two types of calculations described above. A switcher named CCDMOD in this figure selects correct 
outputs depending on CCD mode and is controlled by setting a register. 

[0161] Comp2RGB conversion sub-modules 1016 converts complementary color formatted data to RGB formatted 
data in complementary CCD mode. Especially for G, data from color adjustment and data calculated by conversion 
formula can be blended by 5 types of blending ratio. Actually, the following calculation is executed based on the con- 
version formula: 

R = Ye - Cy + Mg 
G - rG input + ( 1 " r ) (Ye+ Cy - Mg) (r=0 t 1/4,2/4,3/4,1) 



B = Mg - Ye + Cy 

In RGB CCD mode, data from color adjustment sub-module bypass this sub-module. 

[0162] Figure 101 is a block diagram of comp2RGB conversion submodule1016. Three adders, three subtracters, 
and two multipliers are implemented for executing the calculations above. A gain adjuster tor G named "green_ratio" 
in this figure is adjustable by setting a register In RGB CCD mode, a CCDMOD switcherselects off (high in this figure) 
for bypassing this module. 

[0163] RGB gain tor complementary CCD module allows adjustment of white balance by RGB color format even for 
complementary CCD module. This module is also available in RGB CCD mode. 

[0164] Figure 9a is a block diagram of complementary white balance module 1408. One multiplier and clip circuit is 
implemented for this operation. Each gain for RGB is set by a register. 

[0165] Gamma correction modules 1410 execute gamma correction for each color data in RGB color format For 
this operation, prepare in advance three types of data for approximating the gamma curve by four linear segments. 
Those are area, offset and gain shown in Figure 9b. As shown in Figure 14, this module exists for each color so that 
the independent adjustment to RGB may be made. 

[01 66] Figure 9c is a block diagram of gamma correction module 1 41 0. Area detector selects correct gain and offset 
for input data based on area data. The data regarding gain, offset, and area are set in three registers. 
[0167] RGB2YCbCr conversion module 1412 converts RGB formatted data to YCbCr formatted data and adjusts 
offsets to Cb and Cr based on the following matrix calculation. 
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Each coefficient in this matrix is set by a register so that variable setting for this conversion is available. 
[0168] Figure 11 is a block diagram of this RGB2YCbCr conversion module 1412. Nine multipliers and five adders 
are implemented for the foregoing matrix calculation. After multiplying RGB data with coefficients, the six least signif- 
icant bits of each data from the multipliers is cut in order to reduce size of circuits. As to Cb and Cr additional circuit 
for offset adjustment is followed by YCbCr conversion circuit. Clip circuits for Cb and Cr includes conversion circuits 
from two's complement to offset binary. 
[0169] Burst mode compression/decompression engine 

[0170] The preferred embodiment DSC engine includes an improved Burst Capture function with real-time process- 
ing, without compromise in the image resolution as compared to the regular capture mode. The Burst Capture Mode 
is the use of dedicated compression and decompression engine 1 08 for an increased burst capture sequence length. 
A sequence of CCD raw image frames is firsL sLored in SDRAM 160 by using Compression engine 108. Then, as an 
off-line process, the image pipeline of regular capture mode retrieves the CCD raw images from SDRAM 1 60, processes 
them sequentially, and finally stores them back as JPEG files in the SDRAM. The Animated Playback Mode can display 
these JPEG files. 

[0171] Burst mode compression/decompression engine 108 includes differential pulse code modulation (DPCM) and 
Huffman coding using the same tables as the entropy -coding of DC coefficients in baseline JPEG compression. Engine 
108 uses the sample Huffman table in the JPEG standard for chrominance DC differential data. Engine 108 also pro- 
vides the inverse transforms as illustrated in Figure 13. Fixed Huffman Table (JPEG Huffman table for Chrominance 
DC coefficients): 



Category (SSSS) 


A 


Code Length 


Codeword 


0 


0 


2 


00 


1 


-1,1 


2 


01 


2 


-3,-2 : 2 : 3 


2 


10 


3 


-7,. .,-4,4,. ,7 


3 


110 


4 


-15,. .,-8,8,- ...15 


4 


1110 


5 


-31 ,..,-16,16,,. ,31 


5 


11110 


6 


-63,..,-32,32,..,63 


6 


111110 


7 


-127,.. : -64 : 64,. M 127 


7 


11111110 


8 


-255,..,-128 : 128,.,128 


8 


111111110 


9 


-511,. .,-256,256,. .,511 


9 


1111111110 


10 


-1023... ,-512,512,.. ,1023 


10 


11111111110 


11 


-2047,. ,-1 024,1 024,. .,2047 


11 


111111111110 


12 


-4095,. .,-2048, 2048... ,4095 


12 


1111111111110 



The encoder has four look-up tables: Huffman code (13 x 2-byte entries).. Huffman code length table {13 x 1-byte 
entries), low bit mask to generate variable-length bit stream {32 x 4-byte entries), nad log table (256 x 1-byte entries). 
The Huffman tables are not programmable for simplicity, although alternative embodiments could include programma- 
ble Huffman tables. 

[0172] The Huffman decoder performs the inverse function of the Huffman encoder and has five look-up tables: max 
code comparison table (1 3 x 2-byte entries), Min code comparison table (1 3 x 2byte entries), decoded Huffman symbol 
pointer (1 3 x 1-byte entries), decoded Huffman symbol table (13 x 1-byte entries), and bit position mask (32 x 4-byte 
entries). 

[0173] The lossy mode compression just discards the least significant bit (LSB) or the two least significant bits of 
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each coefficient. 
Playback synchronization 

[0174] A problem invoked in playback of audiovisual bitstreams is how to synchronize audio with video signal. The 
preferred embodiments play the audio bitstream seamlessly in the background in real-time with the audio encoded by 
using the simple coding standards like ITU-T G.711 and Microsoft 16-bit PCM. By using an interrupt service routine, 
about 0,1% of the DSP resources is enough to output audio in real time through {multichannel) buffered serial ports; 
see Figure 1b. Therefore, the preferred embodiment must realize the video decoding in synchronization to the audio 
playback. 

[01 75] For clarity, assume that both audio and video are captured in full speed (real-time with 8K sample/s for audio 
and 30 frame/s for video). Audio is played back as samples. However, video is displayed in the granularity of frames. 
Thus the synchronization problem is caused by the fact that the video decoding could be faster or slower than the real- 
time requirement. If the video decoding Is too fast, a certain amount of delay slots has to be inserted to slow down the 
decoding. Contrarily, if the video decoding is too slow f some video frames must be skipped to catch up with the real- 
time audio playback. 

[0176] The preferred embodiments handle both cases. Especially in the case of slow video decoding, the preferred 
embodiments can properly select and skip the frames in an optimal manner Note that the preferred embodiment is 
described for video bitstreams without bi-directional coded frames (B-frames). 

[0177] Figure 46a depicts the synchronization between audio and video. The first video frame is pre-decoded before 
beginning audio-video playback. Since the video is displayed in the granularity of frames, the synchronization points 
are located at the video frame boundaries, i.e. (t = 0 : AT 2AT, 3AT....). Here AT is the duration of a frame, which is 
defined as: 

AT=Vfp (!) 
where fp is the frame-rate used for the video sequence. 

[0178] Audio and video could lose synchronization when the video decoding speed is not fast enough. As illustrated 
in Figure 46a, when the decoding of video frame 2 has not finished in time (Td2 > AT), the audio-video playback loses 

synchronization after displaying video frame 1 . Here {Tdm, m =0, 1 ,2, } denotes the decoding time used for decoding 

video frame m. 

[01 79] With insufficient video playback speed, the only way to maintain a reasonable synchronization between audio 
and video is to skip video frames properly. In Figure 46b, video frame 2 is skipped (and frame 1 repeated) so that 
synchronization can be reacquired at frame 3. 

[0180] A preferred embodiment circular buffer scheme is illustrated in Figure 47. The video decoder is connected to 
one side of the circular buffer, the display is connected to the other side. The circular buffer has a size of N video 
frames. There are two registers associated with each frame buffer of the circular buffer: the first register contains TP n , 

n=0, 1 , 2. 3 , N-1 which indicates the presumptive presentation time of the video frame stored in buffer n, and the 

second register contains S n , n=0, 1,2,3, , N-1 which signals whether the frame in buffer n is ready for display (1 

for ready, 0 for not ready). Of course J the value of TP n is a multiple of AT Buffer switching for display also occurs at 
frame boundaries (i.e. at timet = mAT,m=0, 1,2,...). Because the preferred embodiments use a circular buffer containing 
N frames, all the indices (..., n-1 , n, n+1 , ...) should be regarded as modulo-N indices. 

[0181] Suppose the time after decoding the current video frame is T The decoded current frame is stored in buffer 

n-1 in Figure 47. Therefore, the buffer to be used for storing the next frame in Figure 47 is buffer n. 

[0182] Determine the current position in the bitstream: the frame index m of the current decoded frame is defined as 

m=TP^/M (2) 

Determe the decoding starting time of the next frame: since the frame in the buffer n is to be displayed during the time 
interval of {TP n < t < TP n+1 ), buffer n is not available for decoding the next frame until TP n+1 . Therefore, the decoding 
starting time of the next frame Ts is: 

rs=max{TTF„ +7 } (3) 
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Determine the next frame to be decoded: Jet Tdbe the estimated time for decoding the next frame., the presentation 
time of the next frame must satisfy; 



ft 



TP n >Ts+Td 



The above conditions imply that the decoding of the next frame is finished before its presentation time, and the next 
frame is located at least a frame after the current frame in the bitstream. 

Because TP n must be a multiple of AT, the next frame that can be synchronized to audio satisfies the conditions: 



TP=LT 



AT 

IP*77L, +AT 



Ts*Td 



+0.5 



where [•] denotes integer part by truncation. 

[01 83] Therefore, the presentation time of the next frame is determined by: 



TP. = max \ AT 



Ts +Td 
AT 



-i- 0.5 



(4) 



There are different methods to estimate Ttf, such as using statistical estimation based on prior decodings or frame 
parameters. One preferred embodiment simply uses the actual decoding time of the most recently decoded frame of 
the same picture coding type (l-frame or P-frame) plus a certain amount of safety margin as the estimated decoding 
time for the next frame. 

[0184] The frame index m' of the next frame to be decoded can thus be computed as: 



Then the number of frames Am to be skipped from the current position is determined by: 



(5) 



4m=m'-m-1 



(6) 



Equation (2) to (6) make up of the basic control operations for updating the circular buffer. 

[01 85] The preferred embodiments use the circular buffer scheme to realize synchronization. There are two parts: 
the video decoder buffer switch control and the display buffer switch control. Figure 48 demonstrates the flowchart of 
the video decoder buffer switch control, which contains two stages: initialization and playback. 
[0186] Initialization: in the circular buffer initialization, N f (is N, <N) video frames are decoded before starting play- 
back. As shown in the dashed box in Figure 48 : there are four steps for the initialization: 

step 0: set all the presentation time registers {TP n , n=0, 1 , 2, 3, N-1 } and the status registers {S n ,n=0, 1 , 2, 

3, , N-1] to zero, switch the video decoder to buffer 0 (i.e. n=0), point to the beginning of the video bitstream 

(i.e. m' = Am = 0), set time to zero (i.e. t =0) 

step 1 : set the related status register S n to 1 , skip Am video frames, decode frame m* : store the decoded frame in 
buffer n. (Recall on the first pass through the loop, n=0, m'=0, so the first frame is decoded and stored in buffer 0.) 
step 2: set the decoding start time Ts to t ( switch to the next buffer (i.e. n++), update TP n , m', Am according to 
equations (4), (5), and (6). 
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step 3: check whether the number of decoded frames reaches the pre-set frame number N f . If true, go to playback, 
otherwise, loop to step '1 . 

[0187] Playback: there are six steps involved in updating the circular buffer during the playback. 

step 0: switch display to buffer 0, enable display reset time to zero (i.e. t = T=0), switch the video decoder to buffer 
N f (i.e. n = N f ) 

step 1 : if the whole video sequence is decoded, stop decoding, otherwise, go to step 2. 
step 2: update Ts ; TP n , m' and Am according to equations (3) ; (4), (5) t and (6). 
step 3: wait until time reaches Ts (i.e. t > Ts), go to step 4. 

step 4: set the related status register S n to 0, skip Am video frames, decode frame m', store the decoded frame in 
buffer n. 

step 5: if the frame decoding finishes in time (i.e. t < TP n ), set S n to 1 to indicate the decoded frame is ready for 
display, set^T to t^switch the video decoder to the next buffer (i.e. n++). Otherwise, set T to t, add DT to the 
estimated Td(\,e, Td-h = DTwith DT= N^T, intentionally skip N d (0< N d ) more frames in the next stage), set the 
current frame index m to m\ Go to step 1 . Note that N d is a parameter to control the screen freezing rime before 
resuming the synchronization. 

[0188] Users can freely decide the circular buffer size (N) : the iniLial lime delay (N f ) for the playback as well as Lhe 
screen freezing time (Nd). Obviously, the minimum buffer size is 3 video frames (i.e. N = 3), the least time delay is one 
video frame (i.e. N f - 1). However, in the case of insufficient video decoding speed, ft is strongly recommended to 
decode N-1 frames (i.e. N f = N-1 ) during the circular buffer initialization, so that the video decoder can gain the maximal 
room to catch up with the audio real time playback. 

[0189] Display buffer switch control: the display buffer switch control is carried out in parallel to the video decoder 
buffer switch. The preferred embodiment checks the display buffer switch at video frame boundaries: t = rnAT, m=0, 
1,2,.... Suppose the display is currently showing the video frame in buffer n-1 , it switches to the next buffer, i.e. buffer 
n, if and only if the current time (t>TP n ) and (S n =1) holds. Otherwise, it is connected to buffer n-1 . Here., if (teTPJ and 
(S n =0), it means the decoder has not finished decoding of the frame in time. In this case, the video frame in buffer n 
has been discarded, the decoder is decoding the conservatively selected next frame to update buffer n again, the 
display should keep displaying the frame in buffer n-1 until (t>TP n ) and (S n =1) holds. 

[0190] In summary, the preferred embodiment provides a way to realize the synchronization between audio and 
video when playing back by using software or firmware. 

Variable length decoding 

[0191] Variable Length Decoding (VLD) is involved in decoding bitstreams which are generated by using Variable 
Length Encoding (VLC) at encoder; see Figure 1b item 126. Because of VLC, the number of bits used for coding units 
varies from unit to unit. Therefore, a decoder does not know the number of bits used for a coding unit before having 
decoded it. This makes it essential for a decoder to use a bitstream buffer during the decoding process. 
[0192] In video coding, for example, a frame to be encoded is decomposed into a set of macrob locks (see Figure 
49). Under the consideration of the smallest memory requirement, a coding unit here is normally defined as macroblock, 
which consists of a 1 6x1 6 pixel luminance area and the corresponding chrominance areas depending on the chroma 
format (4:2:0, 4:2:2:, or 4:4:4) . Certainly, a slice (a row of macroblocks in a frame) or even the frame itself can be 
treated a coding unit if there is enough memory. 

[0193] Figure 50 depicts the video playback on a preferred embodiment digital still camera (DSC). In DSC applica- 
tions, the video bitstream is pre-captured and stored on the high-capaciry SDRAM, and the video decoder is built on 
the DSP. Since it is extremely expensive for the decoder to directly access Lhe SDRAM, an on-chip bitstream buffer is 
opened on the DSP internal memory. The bitstream is first loaded from SDRAM to the bitstream buffer through the 
SDRAM, then the decoder uses the bitstream in the bitstream buffer to reconstruct video. Since the bitstream loading 
is achieved by using DMA (Direct Memory Access), which can run in the background without intervention of a CPU, 
the bitstream loading overhead is mainly due to time used for setting up registers for the DMA transfer. 
[0194] There are two basic requirements in terms of bitstream buffer management. First of all, the buffer size should 
be big enough to cover the worst case. For example, in video coding, the theoretically maximaJ number of bits for 
encoding a macroblock could be 25S words (one word here is defined as two bytes). Although this worst case is very 
rare, the bitstream buffer size has to be 258 words in order to be at the safe side. Secondly, the bitstream buffer should 
never underflow, that is, the buffer management should guarantee that the bitstream for a coding unit is available when 
it is being decoded. 

[0195] There are different schemes to satisfy the second requirement. The simplest one would be to check the 
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decoding position in the bitstream buffer at each buffer access . The bitstream buffer is re-filfed whenever the decoding 
position is out of the valid buffer range. Because the decoding is a bit by bit operation, this scheme is not realistic: it 
spends too much overhead in deciding when to re-fili the buffer. 

[0196] A realistic scheme is the linear shifting buffer scheme as shown in Figure 51a. In this scheme, the bitstream 
buffer is linearly accessed by the decoder from left to right, after decoding a unit the rest of the bitstream is shifted 
forward to the beginning of the buffer, then the buffer is re-fiiled to "fuir before decoding the next unit. In Figure 51a. 
Ps and Pd denote the current decoding position and the bitstream end position in the bitstream buffer, respectively. 
[0197] This bufler scheme has rwo disadvantages. First, since the buffer size is much larger than the average number 
of bits of the decoding units, a lot of time will be spent on the bitstream shifting. For instance, in video decoding the 
buffer size is 256 words to cover the worst case : but on average a unit may only use 1 6 words, this means about 240 
words of shifting for each unit. The second disadvantage is that it requires a bitstream loading after decoding each 
unit; this costs additional overhead because time has to spent on issuing the DMA transfers. 
[0198] A better buffer management scheme is so-called quasi-circular buffer scheme as shown in Figure 51b. In this 
scheme, the decoder accesses the bitstream buffer in a circular manner. This avoids the bitstream shifting required 
by the linear buffer scheme. There are two cases after decoding a unit. This first case is in the lefthand portion of Figure 
51 b: the rest of bitstream is located in the middle of the buffer. In this case, the buffer is filled by loading the bitstream 
twice, one for the right end followed by the other one for loading the left end, (Note: if the bitstream loading can write 
the bitstream into the bitstream buffer in a circular manner, only one load is needed; however, this is not always the 
case.) The second case is shown in the righlhand portion of Figure 51 b, in which only the middle of the buffer needs 
to be filled. 

[0199] The quasi-circular buffer scheme is much more efficient than the linear shifting buffer because it avoids bit- 
stream shifting, but it still suffers from a disadvantage that one or two bitstream loads are needed after decoding each 
unit. The following preferred embodiment hybrid circular-double buffer scheme solves this problem. 
[0200] Figure 52 status 0 shows a hybrid circular-double buffer containing two buffers of equal size; namely, the left 
buffer and the right buffer. There is a flag for each buffer to indicate the buffer fullness ("full" / "not-full"). Ps points to 
the current decoding position after decoding a unit. In terms of buffer size, each buffer covers the worst case of decoding 
coding units, this makes the hybrid buffer size twice of a linear shifting buffer or a quasi-circular buffer. Unlike a traditional 
double buffer, the two buffers here have a continual memory allocation, i.e. the left buffer is directly followed by the 
right buffer in the memory map. The decoder accesses the hybrid buffer in a circular manner 
[0201] The preferred embodiment hybrid buffer operates through the following four statuses: 
[0202] • Status 0: the initialization status, both the left and right buffers are fully loaded and set to "full", Ps points to 
the beginning of the hybrid buffer. 

Status 1 : after decoding the first unit., change the left buffer flag to "not-fufr. 

Status 2: after decoding a unit, if the current decoding position Ps is in the right buffer and the left buffer flag is 
"not-full", fully load the left buffer and set the left buffer flag to "full", in addition, if the right buffer flag is "full"., change 
it to "not -full". Otherwise, no aclion is taken. 

Status 3: after decoding a unit, if the current decoding position Ps is in the left buffer and the right buffer flag is 
"not-full", fully load the right buffer and set the right bufter flag to "full". If the left buffer flag is "full", change it to 
"not-full". Otherwise, no action is taken. 

[0203] Taking the preferred embodiment platform (e.g. Figure 1b) as an example (where data is in 16-bit units), 
define the following data type: 



typedef struct bitstream { 

SInt bit_ptr; /* current bit position (0 - 16) 

SInt Ps; /* current decoding position in 

bitstream buffer */ 
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SInt left_£lag 



/* left buffer flag "full / not- 



full- 



SInt right_flag 



right buffer flag "full / not^ 



full" 



USInt *databuf; 



/* bitstream buffer 



Long Addr_SDRAM; /* bit stream address in SDRAM 



} Bitstream; 



The ps§udo code shown in Table 1 . describes the hybrid circular-double buffer scheme. Function Buffer! niliali2ation() 
is called only once at the beginning of decoding., while function Bits treamBufferUpd ate () is called after decoding each 
coding unit, it automatically updates the buffer flags and re-loads the buffers if the conditions become true. In Table 1 
BUFSIZE stands for the buffer size of the hybrid circular-double buffer. 



Void 3uf f erlnitialization ( 



Bitstream *stream, /* pointer of bitstream 




*/ 



/* Initialization of the hybrid circular -double buffer 




LoadBuf f er (&stream->databuf [0 j , 



s t ream- > Addr_SDRAM , 



BUFSIZE) ; 



stream->Addr_SDARM +=BUFSIZE; 
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stream->lef t_f lag = n full 11 ; 
stream- >righc_f lag = " full* 
stream->Ps - 0 ; 
stream- >bit_ptr = 16; 



10 



75 



20 



Void BitstreamBuf f erUpdate( 

Bit stream *stream, /* pointer of bitstream 



5 



30 



35 



40 



= = = = = = = 

/* Update the left buffer if necessary 



if (stream->lef t_f lag ==" not- full" stream->Ps >= 

EUFSIZE/2) 

{ * 

LoadBuf f er (&streaiu->databuf [0] , stream- >Addr_SDRAM, 
BUFSIZE/2); 

stream->Addr_SDARM +=BUFSlZE/2; 
streaiu->lef t_f lag - "full"; 

> 
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/* Update the right buffer if necessary 



7 



if (stream->right_£lag =="not-f uil" uSl stream->Ps < 
BUF3IZE/2) 

{ 

LoadBuf fer (&stream->databuf [BUFSIZE/2] , stream- 
> Addr__SDRAM , BUFSIZE/2) ; 

stream->Addr_SDARM +=BUFSIZE/2 ; 
stream->right_f lag = "full"; 



stream->Ps < 



/* Update the left buffer flag 

if (stream~>lef t — f lag =="full" 
BUFSIZE/2) 

stream->lef t_f lag = "not-full n ; 



/* Update the right buffer flag 



if ( stream~>r ight_f lag =="full' 
BUFSIZE/2) 



streain->Ps >= 
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stream- >right_f lag * "not-full " ,- 



} 



Table 1. Pseudo code for che hybrid circular-double buffer 
scheme 

As it can be seen in BitstreamBufferUpdate() in Table 1, the left buffer or right buffer is not reloaded after decoding 
each unit., but is loaded only if the opposite buffer (left / right) is in use and its buffer flag is 'not-full". This greatly reduces 
the numberor buffer loads. Consider the video coding as an example. This needs BU FSIZE or 5 12 words if a macroblock 
is the unit : the average bitstream size of a unit is assumed to be 16 words. Because the linear shifting buffer and the 
quasi-circular buffer re-fill the buffer after decoding each unit the average loading length for those two schemes is also 
1 6 words. Compared with thefixed loading length of 256 words inthe hybrid circular-double buffer scheme., the preferred 
embodiment reduces the loading overhead by a factor about 16 (i.e. 256/ 16). 

[0204] Mini-experiments compared the three buffer schemes discussed above. The video sequence used was coast- 
guard (352 x 288, 300 frames, 4:2:0). The bitstream is generated by using a MPEG1 video encoder. The target bit- 
rate is 3 Mbit/s ; l-frame only. The same decoder with three different buffer schemes are used to decode the same 
bitstream, the buffer loading count and word shifting count are recorded during the decoding. The performance com- 
parison among the three buffer schemes is listed in Table 2. As shown in Table 2. for each macroblock the linear shifting 
buffer scheme requires one buffer load, and on average about 240 words of shifting. The quasi-circular buffer scheme 
needs slightly more buffer loads ( 1.06 load/macroblock) but no shifting. The preferred embodiment hybrid circular- 
double buffer scheme used only about 0.0619 buffer load per macroblock. On the preferred embodiment platform of 
Figure 1b in particular, the preferred embodiment scheme provides a cycle count reduction ratio of about 113 and 17 
in comparison to the linear shifting buffer scheme and the quasi-circular buffer scheme, respectively. 



Table 2. 



Performance comparison among three buffer schemes on TMS320DSC21 platform 




Linear shifting buffer 


Qu-circular buffer 


Hybrid circular-double buffer 


Buffer size (words) 


256 


256 


512 


Number of loads per macroblock 


1.00 


1.06 


0.0619 


Number of word shifting per 
macroblock 


240.15 


0 


0 


Overhead per toad (cycles) 


80 


80 


80 


Cycle count per word shifting 


2 


2 


2 


Total cycles used for bitstream buffer 
per macroblock 


560.30 


84.72 


4.95 


Cycle count ratio vs. the hybrid 
circular-double buffer scheme 


113.19 


17.12 


1.00 



Onscreen display and graphics acceleration 



[0205] The Onscreen display (OSD) module 105 is responsible for managing OSD data from different OSD windows 
and blending it with the video. It reads OSD data from SDRAM 160 ; and outputs to MTSC/PAL encoder 106. The OSD 
module defaults to standby mode, in which it simply sends video to NTSC/PAL encoder 106. After being configured 
and activated by ARM CPU 130. the OSD module reads OSD data and mixes it with the video output. ARM CPU 130 
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is responsible for turning on and off OSD operations and writing the OSD data to the SDRAM. Figure 15 shows the 
block diagram of the OSD module and affiliated other items. The various functions of the OSD are described in the 
following paragraphs. 

[0206] OSD data storage. The OSD data has variable size. In the bitmap window, each pixef can be 1 . 2. 4, or 8 bits 
wide. In the YCrCb 4:2:2 window, it takes 8-bit per components, and the components are arranged according to 4:2: 
2 (Cb/Y/Cr/Y ...) format. In the case where RGB graphics data needs to be used as OSD : the application should perform 
software conversion to Y/Cr/Cb before storing it. The OSD data is always packed into 32-bit words and left justified. 
Starting from the upper left corner of the OSD window, all data will be packed into adjacent 32-bit words. 
[0207] Setting up an OSD window. An OSD window is defined by its attributes. Besides storing OSD data for a 
window into SDRAM by ARM CPU 130, the application program also needs to update window attributes and other 
setup in the OSD module as described in the following subsections. 

[0208] Location register. The Location register contains X and Y locations of the upper left and lower right comers 
of each window. The application program needs to set up the CAM and enable selected OSD windows; see Figure 16. 
[0209] Color look up tables. The OSD has the fixed 256-entry cotor look up table (CLUT). The CLUT is used to 
convert bitmap data into Y/Cr/Cb components. In the case of 1 .2 or 4 bitmap pixels., the CLUT can be determined by 
CLUT registers. 

[0210] Blending and transparency. Color biending on the pixel level is also supported. This feature is available for 
the bitmap displays only (Windowl ,2). if the window color blending is enabled, the amount of blending of each pixel 
is determined by the blending factor. As shown in the following table, Ihe window blending supports 5 different levels, 
according to the selected blending factor. The hardware also supports a transparency mode with bitmap. If transparency 
is enabled, then any pixel on the bitmap display that has a value of 0 will allow video to be displayed. Essentially, 
O-valued pixels are considered the transparent color i.e. the background color will show through the bitmap. The Table 
shows the connection between transparency and blending on the same window. 



Transparency 


Blend Factor 


OSD window contribution 


Video contribution 


OFF 


0 


0 


1 




1 


1/4 


% 




2 


1/2 


1/2 




3 


3/4 


1/4 




4 


1 


0 


ON 


0 


if pixel value = 0 


if pixel value = 0 




1 


0 


1 




2 


1/4 


3/4 




3 


1/2 


1/2 




4 


3/4 


1/4 






1 


0 



[021 1] Hardware cursor. A rectangular shape is provided using hardware windowl. With windowl, the cursor always 
appears on top of other OSD Windows. The user can specify the size, color of the shape. When hardware windowl is 
designated as the cursor, only two windows are available for the OSD application. If a hardware cursor is not used, 
then the application can use windowl as a regular hardware window. Figure 17 shows an example of the hardware 
cursor. 

DSP subsystem 

[0212] The DSP subsystem consists of C54x DSP : local memory blocks, iMX and VLC accelerators, shared image 
buffers, and the multiplexers implementing the sharing. 

[0213] C54x is a high performance, low power, and market proven DSP. cDSP hardware and software development 
tools for C54x are also very mature. 

[0214] The DSP carries out auto exposure, auto focus, auto whltebalancing (AE/AF/AWB) and part of the image 
pipeline tasks. It also handles SDRAM transfer and drives the accelerators to implement the rest of image processing 
and image compression tasks. Flexibility and ease of programming in the DSP enables camera makers to refine the 
image processing flow, adjust qualityperformance tradeoffs, and introduce additional features to the camera. 
[0215] The configurable DSP (cDSP) design flow is adopted to allow flexibility and design reuse. The memory blocks 
time-shared among DSP and accelerators are large enough for one processing unit (16x16 pixels) and provide zero- 
wait state access to DSP. 
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Features 

Fixed-point Digital Signal Processor 
100 MIPS LEAD2.0 CPU 
On-module RAM 32Kx16bit 

(4 blocks of 8Kxl6bit dual access program/data RAM) 

Multi-Channel Buffered Serial Ports (McBSPs) 

ARM can access RAM via Enhanced 8-bit Host Port Interface 

One hardware timer 

On-chip Programmable PLL 

Software Programmable Wait- State Generator 

Scan-based emulation and JTAG boundary scan logic 

[0216] Figure 18a shows more details on the DSP subsystem and in particular the details of the connection between 
the DSP and the iMX and VLC. Figure 1 8b is the memory map. 

[021 7] The shared memory blocks A and B occupy two 2Kword banks on the DSP's data memory space. Each block 
can be accessed by DSP iMX, VLC, and SDRAM controller depending on static switching controlled by DSP. No 
dynamic cycle-by-cycle, memory arbitration is planned. DSP's program should get seamless access of these memory 
blocks through zero- wait-state external memory interface. 

[021 8] The configuration memory blocks, for iMX coefficient, iMX command, VLC Q-matrix, and VLC Huffman table, 
also connect to DSP's external memory interface. They are also statically switched between the specific module and 
DSP Typically at powerup or at initial stage of camera operation mode, these memory blocks are switched to DSP 
side so DSP can set up the appropriate configuration information for the operation. Then, they are switched over to 
iMX and VLC for the duration of operation. 

Imaging Extension (iMX) 

[0219] iMX, imaging extension is a parallel MAC engine with flexible control and memory interface for extending 
image processing performance of programmable DSPs. iMX is conceived to work well in a shared memory configuration 
with a DSP processor, such that flexibility, memory utilisation, and ease of programming are achieved. The architecture 
covers generic 1-D and 2-D FIR filtering, array scalingyadditlon, matrix multiplications (for color space transform), clip- 
ping, and thresholding operations. 

[0220] For digital still cameras, iMX can be used to speed up 

CFA interpolation, 
color space conversion, 
chroma down-sampling, 
edge enhancement, 
color suppression, 
DCT and IDCT, 
Table lookup. 

[0221] iMX methodology originates from the discipline of parallel processing and high performance computer archi- 
tecture. The design comprehends the need for a scalable MAC engine. iMX in the first preferred embodiment incor- 
porates 4 MAC units: see Figure 19. Alternative preferred embodiments upgrade to 8 MAC units or more. Software 
can be structured so that the hardware upgrade will not incur substantial software changes. 

[0222] Much flexibility of iMX is due to parameter- driven address generation and looping control. Overall efficiency 
comes from efficient pipelining control inside iMX as well as the system-level memory buffering scheme. 
[0223] iMX works best for block-based processing. To facilitate this, the datapath needs to connect to data input/ 
output and coefficient memory. iMX contains data inpuL data output, and coefficient memory ports, and allows arbitra- 
tion among these ports. This eliminates the need for dedicated memory blocks, and brings more flexibility and better 
memory utilization on the system level. These memory blocks are accessible as DSP data memory to facilitate data 
exchange. 

[0224] There is a separate command memory that feeds a command decode unit in iMX. The command memory 
should be specified to fit all the accelerated steps in our reference image pipeline algorithm, so that this sequence of 
commands can be executed with little intervention from DSP. 

[0225] iMX block diagram appears in Figure 20. A command decode subblock reads and decodes commands, and 
drives static parameters, one set per command, to the address generator. Address generator then computes looping 
variables and data/coefficient/output pointers, and coordinates with execution control, which handles cycle-by-cycle 
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pipelining control Address generator sends data and coefficient read requests to the arbiter. Arbiter forwards the re- 
quests to the data/coefficient memory. Data read back from memory go to the input formatter, which takes care of data 
alignment and replication. Formatted data and coefficients are then prodded to the datapath, which mainly consists 
of the 4 MAC uniis. Output from datapath is routed to arbiter for memcr,* write. 

[0226] iMX communicates to DSP via shared memory (for data input, coefficient, data output, command) and via 
memory-mapped registers (start command, completion status). All data buffers and memory blocks are single-ported, 
and are switched to one party or another via static control rather than on-line arbitration. 

[0227] In a typical application, DSP would place filter coefficients, DCT/IDCT cosine constants, and lookup tables in 
the coefficient memory, and put iMX commands in the command memory. DSP then turns over access to these memory 
blocks to iMX, These memory blocks are sized adequately for our reference design to fit all needed coefficients and 
commands for a major camera operation mode (e.g., image capture). Any update/reload should occurvery infrequently. 
In case either or both memory blocks run out of space, paging can be performed. 

[0228] DSP manages the switch network so that, to iMX : there is only one data buffer. During run time, DSP switched 
the A/B buffers among itself, iMX. VLC, and SDRAM controller to implement data passing. 

[0229] Figure 21 illustrates a simple table lookup accelerator with input rounding/clipping capability used to speed 
up the image pipeline on the DSP. This is carried out with a very simple control structure and datapath. 

VLC engine 

[0230] VLC accelerator is a coprocessor optimized for quantization and Huffman encode in the context of JPEG 
compression and MPEG compression. It operates with quantizer matrices and Huffman tables preloaded by DSP, via 
shared memory blocks. Aggressive pipelining in the design achieves very high throughput rate, above 30 million DCT 
coefficients for compression. 

[0231] VLC's working memory : including quantizer matrices, Huffman tables, and data inputfoutput memory, are all 
shared memory blocks. 

VLC functionality 

[0232] Basically, VLC covers Quantization, zigzag scan, and Huffman encode for JPEG encode (baseline DCT, 8-bit 
sample), with up to 4 quantizer matrices (stored as invqfi.j] = 2 16 /q[i,j]) and 2 encode Huffman tables all loadable. Can 
process one MCU that contains up to 10 blocks. Each block consists of 8x8 = 64 samples. 

[0233] Quantization, zigzag scan, and Huffman encode for MPEG-1 video encode. One macroblock, with up to six 
8x8 blocks : can be processed. Number of blocks and within them, number of luminance blocks, can be specified. 
Huffman encode can be bypassed to produce quantized and zigzag-ordered levels. - 

[0234] The accelerator requires memory blocks for input/output buffer, quantization matrices and Huffman encode 
tables. The memory configuration should be sufficient to support normal encode operations, one JPEG MCU (minimum 
coding unit), or MPEG macroblock per call. 

[0235] Both input and output mustf it the 2K words (lword = 16-bit) shared memory buffer(AorB). MCU ormacroblock 
has maximally ten 8x8 blocks, or 640 input words. Compressed output data is typically smaller than input size. 
[0236] JPEG Huffman encode table takes up (12 x 176) x 32-bit, or 384 words per table. JPEG standard allows 2 
tables, so taking totally 768 memory words. MPEG tables are hard-wired into VLC and do not take up memory. We 
have allocated 2K words for the Huffman tables. 

[0237] The quantizer matrix memory, 512 words by 1 6-bit allow for 8 quantizer matrices to coexist, each taking 64 
x 16-bit. JPEG allows for 4 matrices, and MPEG encode requires 2 matrices. 

Figure 22 shows the major subblocks of VLC. Only the encode path is implemented in one preferred embodiment VLC 
module; alternative preferred embodiments incorporate the decode path into the module. 

ARM subsystem 

[0238] ARM microprocessor 130 handles system-level initialization, configuration, user interface, user command 
execution, connectivity functions, and overall system control. ARM 130 has a larger memory space, better context 
switching capability., and is thus more suitable for complex, multi-tasking, and genera] processing than DSP 122. Pre- 
ferred embodiments integrate an ARM7 cTDMI core; see Figure 1 b. ARM7 core is specified up to at least 40 MHz. The 
ARM subsystem will also have a 32 Kbytes local static RAM 132. 

[0239] ARM processor 130 is connected to all the DSC peripherals including CCD Controller, TV encoder, preview 
engine, IrDA, USB, Compact Flash/Smart Media, UART, etc. 

[0240] ARM processor 130 is involved with the management of CCD incoming raw data and intermediate data to 
the SDRAM and LCD. Connected to all I/O devices, the ARM manages and is responsible for the smart devices such 
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as USB t IrDA. Compact Flash/Smart Media, and UARTS. The four basic operation modes of PREVIEW, CAPTURE. 
PLAYBACK, and BURST are initialed by requests from the ARM. The ARM wili then monitor the device for completion 
of the request and in some cases will manage data after the request is completed. 

[0241 J After RESET and before any of the camera operations can occur, the ARM must perform several housekeeping 
5 tasks. The intiai task is known as the BOOT operation task. This function not only iniiializes the I/O and peripherals to 
a known state, it also must prepare, load and start DSP 122. This sequence begins by reading the DSP boot code 
from the flash, loading the DSP code memory and then releasing the DSP from its HOLD state. Additional DSP code 
is loaded into the SDRAM in a format the DSP can then read and overlay into its code space without ARM intervention. 

io ARM SDRAM Interface 

[0242] ARM has two types of access to the SDRAM (1 ) through SDRAM buffer (burst read/write) and (2) direct access 
to the SDRAM with a higher latency - 4 cycle READ, 6 cycle WRfTE. The direct access to memory can be word, half 
word or byte access. 

75 [0243] The ARM/SDRAM controller interf ace also has a 32 byte buffer. The SDRAM burst request first fills this buffer 
and ARM reads and writes from/to this buffer. 

ARM External Memory Interface 

[0244] ARM 130 connects to the external memory through the External memory interface module. ARM 130 connects 
to the Compact Flash/Smart media through this interface. ARM 1 30 also connects to the off chip flash memory through 
this interface. DMA block (Figure 1b) enhances the ARM to CF/Smart media transfer. 

ARM/DSP BOOT Sequence 

25 

[0245] The DSP BOOT sequence begins after a power up or after a COLD START. In this state. DSP 122 is in a 
HOLD condition waiting on initialization from ARM 130. The ARM checks DSP status registers to assure the DSP is 
in a HOLD state. The ARM programs the DSP boot code data to the DSP code memory from the FLASH.. The code 
is organized in logical overlays that allow the ARM to select the proper code for the function needed, in this case BOOT 
30 code. 

[0246] The ARM loads the DSP code using the HPI Bridge (HPIB) interface. This interface can be programmed to 
access in either 8- or 1 6-bil width. For BOOT purposes, this will always be a 16bit access. 

[0247] After the code is loaded, the ARM signals the DSP to begin by releasing the HOLD. The DSP then begins its 
reset sequence from an address of DS P 7F80h which is in the DSP RESET vector area. Upon completion of the RESET 
35 sequence, the DSP then branches to DSP FF80h ; which is the beginning of the BOOT program loaded by the ARM. 
[0248] Figure 23a shows the datapaths used in the ARM/DSP boot sequence as well as data ; request and command 
exchanges discussed later. 

Capture Mode 

40 

[0249] ARM 130 programs CCD controller 102 to capture an image. The CCD controller auto transfers the image 
data to SDRAM and interrupts the ARM using IRQ1 when the transfer is complete. The ARM then notifies the DSP the 
RAW picture data is available to crunch. When the processing of the raw data is complete, the DSP signals the ARM 
the taskis finished. 

45 

Preview Mode 

[0250] The CCD will be programmed for a 30 fps high frame rate but reduced resolution vertically. The reconfiguration 
of the CCD and TG (timing generator) will cause the raw picture data to go to preview engine 1 04. The DSP will post 
so process the data in SDRAM and prepare parameters for FOCUS, EXPOSURE and WHITE BALANCE. The ARM is 
signaled by the DSP when new adjustment parameters are ready and those corrections are applied by the ARM. The 
transferring of the correction parameters use the same communication interrupt architecture as previously mentioned 
and are expected to be at the current frame rate. 

55 Burst Mode 

[0251] The burst mode timing is based on the ARM docking the picture rate from application parameters. Similar lo 
a cross between Capture and Preview modes, the ARM programs the CCD for a capture that stores a compressed 
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image into SDRAM through the compression engine. As in Preview mode, the ARM receives adjustment parameters 
from the DSP to make corrections of FOCUS, EXPOSURE and WHITE BALANCE. 

Idle Mode 

5 

[0252] ARM may use an idle mode to receive correction parameters from the DSP during periods preceding other 
camera modes. If not in a power down situation, this time of 10-15 frames will allow the DSP-to-ARM correction loop 
to make auto corrections on FOCUS, EXPOSURE and WHITE BALANCE. This idle mode will simulate Preview mode 
for the purposes of obtaining a stable correction. 

w 

ARM/DSP communication 

[0253] The communication between ARM 130 and DSP 122 is via the HPIB (Host Port Interface Bridge). The HPIB 
physically connects the DSP (a C5409 type DSP) ports and BUSC (BUS Controller) 134. The ARM accesses the DSP 
f5 memory by programming the HPIB : opening a 32k-word window into the DSP memory map. The map contains the 
data structures shared by ihe ARM and DSP for command request's, acknowledgements and datagrams. 
[0254] The HPIB contains five sub-blocks. They are the interface, timing generator, DSP control registers, and in- 
terrupt hold sections. 

[0255] The interface section receives and stores data from BUSC 134 and transfers it to and from the C5409. This 
20 interface can be an B- or 16-bit data path to the C5409 and is 16-bit to the BUSC. An added feature is the ability to 
exchange the upper and lower byte if programmed to do so. 

[0256] The timing generator makes signals HBIL and HDS and detects signal HRDY. HBIL is the HPI byte identifi- 
cation signal- to the C5409. The HDS is the data strobe signal to the C5409 and the HRDY is the ready signal read 
from the C5409. 

25 [0257] The interrupt hold section wilJ detect the HINT level and make the fNTC pulse synchronized with the ARM 
clock. The module will also set the HOLD port of the C5409 and detect HOLDA. 

[0258] In 8-bit mode, address data from the ARM will not reach the C5409. The address is used only if the C5409 
internal memory is selected. Therefore, the ARM must set the address in the HPI A register before sending or receiving 
data to the 32 Kword DARAM. The 8-bit mode may also be used for ARM<->DSP handshaking. The ARM will use the 
30 HINT bit in the HPIC register to interrupt the C5409. 

[0259] in 1 6-bit mode, the HPIA/HPIC/HPID are not used. The ARM can access the C5409 internal memory as if it 
exists in the HPIB module. This mode will deliver faster performance, but does not support the HANDSHAKE signals 
because of these are routed in the HPIC register. 

[0260] Figure 23b shows the signals and paths for the ARM to reach the C5409 DARAM. 
35 [0261] Figure 23c indicates the shared memory map between the ARM (HOST) and the C5409 processor. When 
the ARM selects the memory area, "DSP Memory", BUSC takes cs_hpib signal active. The ARM can now access the 
DSP internal memory (32 kword DARAM + HPIA + HPIC + HPID). 

[0262] When the ARM selects the M DSP Controller area, BUSC takes cs_dspc signal active. The ARM is now ac- 
cessing registers related to the C5409. 

40 

Multi-processing debugging environment 

[0263] The preferred embodiment integrates ARM 130 and DSP 122 and thus multi-processing and thus requires 
debugging and development support. The preferred embodiment accomplishes this with a single JTAG connector 1 70 
with additional emulation logic as illustrated in Figure 24. 

Input/Output modules 

[0264] The input/output module provides the different interfaces with the DSC peripherals as follows, 
so [0265] TV encoder 1 06 produces NTSC/PAL and RGB outputs for the LCD display and TV. 

[0266] CCD/CMOS controller 102 generates timing signals VD/HD, can synchronize on externally generated HD/ 
VD signals (#0 of MODESET register, #0 of SYNCEN register), supports progressive scan and interlaced CCDs, gen- 
erates black clamping ontrol signals, programmable culing pattern 9CULH, CULV registers), 1 line/2 line alternating 
fields, MCLK (generated by CCD module), WEN (WRQ on TG, active-high) indicates CCD controller writing data to 
55 SDRAM, TG serial port interface (elk, data, TG chip select) is controlled by GIO pins, Iris, mechanical shutter, focus 
and zoom are controlled by GIO pins. 

[0267] USB 142 from programmer's perspective consists of three main parts: FIFO controllers, UDC controller, and 
UDC core. USB configuration: INTERFACED0 ALTO ENDPOINT0: CONTROL; INTERFACEO ALTO ENDPOINT1: 
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BULKIN; INTER FACEO ALTO ENDPOINT1 : BULK OUT; INTERFACE1 ALTO ENDP0INT2: ISOIN: INTERFACE2 ALTO 
ENDP0INJT3: INTERRUPT IN. Buffer configuration: SUB module has six FIFOs inside: each FiFO is of the same 
construction, except for direction and buffer size: USB module has only one unified memory for all endpoints: buffer 
sizes are programmable as long as all buffers fit inside the memory. 

[0268] UART part of I/O block 140, supports start/stop communication protocol, detects parity errors (supporting dta 
length of 7 or 8 bits with even, odd, or no parity and 1 or 2 stop bits), has 32 bytes of FIFO for both transmitter and 
receiver generates interrupts for a FIFO overflow or a time-out is detected on data receiving. ARM 130 control UART 
modules. There are seven 1 6-bit width registers which are accessible from ARM 1 30: data transmitter/receiver register 
(FIFO), bit rate register mode register FIFO control register for receiver FIFO control register for transmitter line 
control register, and status register. Figure 25 is a block diagram. 

[0269] Compact Flash/Smart Media interface 130 is used to save/store image or user's data to a compact flash card 

or smart media; see Figure 26. The interface supports two kinds of operation modes for register setting and data 

transfer: memory mapped mode and I/O mode. An ARM 130 interrupt is generated for card detection while a compact 

flash card is being plugged or unplugged. The pins for both the smart media and the compact flash control interfaces 

are overlapped and can be switched by ARM 130 depending on product needs; see Figure 26. 

[0270] In particular, the compact flash controller has registers mapped to the ARM memory space. The compact 

flash controller is responsible for generating the related control signals to the interlace pins, and writes at 420 KB/s 

and reads at 2.0 MB/s. SDRAM can be utilized for storing at least one picture and an attempt to write to the compact 

flash wi.Lh a big sector count, as done in a DOS machine, will invoke the fast write performance. 

[0271] In contrast, the smart media controller has five register settings: command register; addressl register. address2 

register, address3 register, and data port register. These five registers are mapped to the ARM memory space, and 

smart media controller wit generate the related signals for different register access automatically. 

[0272] Audio input/output may be through the serial port of I/O block 140 with DSP buffering. 

[0273] Infrared data access (IrDA) is supported by a fast FIR core and part of I/O block 140. 

[0274] Block 140 also contains general purpose input/output which can support items such as CCD/CMOS imager 

module control for tuning AGC gain and electronic shutter, RTC control, battery power detection which can generate 

Inner interrupt to the ARM for appropriate system response, camera lens motor control for focus and zoom, a user 

keypad input, LED indicators, flash light control, and power management control. 

iMX programming 

[0275] DSP 122 instructs iMX 124 to perform tasks by sending iMX commands. These commands can be complex 
jto understand and contain many parameters that are fixed in the inner loops. The dieal model is to provide separate 
command building and commandtransfer routines to the DSP progarmmer so that the commands can be pre-con- 
structed outside the loop, and transferred to iMX as generaic data memory moves inside the loop. Commonly used 
iMX commands are prepackaged in C code to ease the programming. 

ARM/DSP task allocation 

[0276] ARM 1 30 runs an operating system such as Windows CE : controls low frequency, synchronous input/output 
(such as to a compact flash card (CFC), and controls user interactions which also are slow and all the peripheral 
modules control preview engine, burst mode compression, TV encoder, CCD controller, USB, CF, IrDA, etc. 
[0277] DSP 122 runs an operating system such as SPOX, controls all real-time functions (auto focus, auto exposure, 
auto white balance), real-time input/output (audio IO, modem IO), real-time applications (e.g., audio player), compu- 
tational expensive signal processing tasks (image pipeline, JPEG 2000, Image stitching). 

Pin description of integrated circuit chip 

[0278] The preferred embodiment pins are as follows 



CCD SENSOR 


Pin Count : 16 




1. C_PCLK 

2. C_VSYNC 

3. C_HSYNC 

4. C_FIELD 


(I) Pixel clock 
(I/O) Vertical sync 
(I/O) Horizontal sync 
(I/O) Field indicator 
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(continued) 



CCD SENSOR 


Pin Count : 16 




5. C_WEN 
6:17.C_DATA 


(I) CCDC write enable 

(I) image data 12Bit 



10 


SDRAM Interface 


Pin Count : 58 






1.SDR_CLK 


(0) Master clock 




2.SDRJ3KE 


(0) Clock enable 


15 


3.SDR_WE 


(0) Write enable 




4.SDR_CAS 


(0) Column address strobe 




5.SDR_RAS 


(0) Raw address strobe 




6.SDR_CS0 


(0) Support 2pc of RAM 


20 


7.SDR_CS1 


(0) Support 4pc of RAM 




8:39.DQ[31:0] 


(I/O) Data bus 




40:54.SDR_A[14:0] 


(O) Address bus 




55.SDR_DQMHH 


(O) DQMH for DQ[31:24] 




56.SDR_DQMHL 


(0) DQMH for DQ[23:1 6] 


25 


57.SDR_DQMLH 


(0) DQMH for DQ[1 5:8] 




58.SDR_DQMLL 


(O) DQMH for DQ[7:0] 



ARM BUS 


Pin Count : 39 




1:23.ARM_A[22:0] 
24:39. ARM_D[1 5:0] 


(O) Address bus 
(O) Data bus 



35 





Audio Interface 


Pin Count : 6 




40 


1.DSP_BDX 


(0) Serial port transmit 




2.DSP_BCLKX 


(l/Q) Transmit clock 




3.DSP_BFSX 


(I/O) Frame syncronization pulse 




4.DSP_BDR 


(I) Serial data receive 




5.DSP_BCLKR 


(I) Receive clock 


43 


6.DSP_BFSR receive 


(I) Frame synchronization pulse 



50 


External Rash Interface 


Pin Count : 5 




1.FLSH_WE 


(0) Write enable 




2.FLSH_CE 


(0) Chip select 




3.FLSH_OE 


(0) Output enable 


55 


4.FLSH_SIZE 


(l)8Bit/16Brt select 




5.FLSH_BSY 


(I) Busy input 
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USB(T.B.D) 


Pin Count : 10 


1.M48X0 


(0) 48MHz clock output 


2.M48XI 


(I) 48MHz clock input 


3.USB_DP 


(I/O) Differential data+ 


4.USBJDM 


(I/O) Differential data- 


5. ATTACH 


(I) Attach detect 



UART 


Pin Count 


:5 


1.RXD 


(I) UART RX 


2.TXD 


(0) UART TX 


3.ERXD 


(I) UART Rx for externa! CPU 


4.ETXD 


(0) UART Tx for external CPU 


5.SIFDO 


(0) Serial 1/F data output 



IrDA 


Pin Count : 2 


1.IRXD 


(I) IrDA RX 


2.ITXD 


(0) IrDA TX 



Compact Flash 


Pin Count : 9 


1.CFE1 


(O) Card enable#1 


2.CFE2 


(0) Card enable#2 


3.IOIS16 


(0) I/O select 


4.STSCHG 


(I/O) Status changed 


5.CFWAIT 


(I) Wait signal input 


6.CFRST 


(0) Reset 


7.CFD1 


(I) Card Detect pin#1 


8.CFD2 


(I) Card Detect pin#2 


9.CFRDY 


(I) Ready 



TV/RGB DAC Analog output 


Pin Count : 27 


UREF(R) 


(I) R-ch Current reference control 


2.DAOUT/R) 


(O) Analog output R-ch 


3.GNDA 


Analog GND 


4.VCCA 


Analog VCC 


5.BIAS 


(I) Phase compensation cap. R-ch 


6.VREF 


(I) RGB common reference voltage 


7. IREF(G) 


(I) G-ch Current reference control 


8. DAOUT(G) 


(0) Analog output G-ch 


9. GNDA 


Analog GND 
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(continued) 



TV/RGB DAC Analog output 


Pin Count : 27 


10. VCCA 


Analog VCC 


11. BIAS 


(I) Phase compensation cap.G-ch 


12. IREF(B) 


(I) B-ch Current reference control 


13. DAOUT(B) 


(0) Analog output B-ch 


14. GNDA 


Analog GND 


15. VCCA 


Analog VCC 


16. BIAS 


(I) Phase compensation cap. B-ch 


17. IREF(C) 


(I) Composite Current reference control 


18. DAOUT(C) 


(0) Analog output Composite 


19. GNDA 


Analog GND 


20. VCCA 


Analog VCC 


21.VREF 


(I) Composite reference voltage 


22. BIAS 


(I) Phase compensation cap. composite 


23.DVCC 


Digital VCC for DAC 


24.DGND 


Digital GND for DAC 


25.HSYNC 


(0) H-sync output for RGB output 


26.VCSYNC 


(0) V-sync / Composite-sync(select by register) 



GIO 


Pin Count 


: 32 [31 :0] 


1:32. GIO 


(I/O) General Purpose I/O 



Micellnaeous 


Pin Count : 15 


1. RESET 


(I) Power on reset 


2.M27XI 


(I) 27MHz input 


3.M27XO 


(0) 27MHz output 


4.TCK 


(1) JTAG clock 


5.TDI 


(1) JTAG data input 


6TDO 


(0) JTAG data output 


7.TMS 


(1) JTAG test mode select 


8TRST 


(1) JTAG test reset 


9.EMU0 


(I/O) Emulator interrupt 0 pin 


10.EMU1 


(I/O) Emulator interrupt 1 pin 


11TESTO 


(I) Test input 0 


12.TEST1 


(I) Test input 1 


13.SCAN 


(I) Test input 


14. TESTS LO 


(I) Test mode select 0 


15.TESTSL1 


(I) Test mode select 1 



TOTAL PIN COUNT 


CCD SENSOR 
SDRAM l/F 


58 


17 
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(continued) 



TOTAL PIN COUNT 


ARM BUS 


39 




Audio l/F 


6 




Flash memory l/F 




5 


USB 


5 




UART 


5 




IrDA 


2 




Compact Flash l/F 




9 


4DAC 


26 




GIO 


32 




Micellnaeous 




15 


Sub Total 




219 pins 


Power : 


37 pins (14%) 




TOTAL 


: 256 pins 




Audio player 







[0279] Portable digital audio players are expected to be one of the most popular consumer products. Currently the 
MP-3 player based on MPEG-1 Layer 3 ausio compression standard is growing rapidly in portable audio market while 
MPEG-2 AAC and Doby AC-3 are alternative digital audio coding formats to be considered as emerging standards. 
Thus the preferred embodiments's programmability permits inclusion of digital audio player functions. The audio can 
be input via flash memory, PC. etc. and the decoded can be output on the serial port. The decoding program can be 
loaded from flash memory, ROM, etc. 

[0280] Insofar as embodiments of the invention described above areimplementable, at least in part, using a software- 
controlled programmable processing device such as a Digital Signal Processor microprocessor, other processing de- 
vices, data processing apparatus or computer system, it will be appreciated that a computer program for configuring 
a programmable device, apparatus or system to implement the foregoing described methods is envisaged as an aspect 
of the present invention. The computer program may be embodied as source code and undergo compilation for imple- 
mentation on a processing device, apparatus or system, or may be embodied as object code, for example. The skilled 
person would readily understand thatthe term computer in its most general sense encompasses programmable devices 
such as referred to above, and data processing apparatus and computer systems. 

[0281 ] Suitably, the computer program is stored on a carrier medium in machine or device readable form, for example 
in solidstate memory or magnetic memory such as disc or tape and the processing device utilises the program or a 
part thereof to configure it for operation. The computer program may be supplied from a remote source embodied in 
a communications medium such as an electronic signal, radio frequency carrier wave or optical carrier wave. Such 
carrier media are also envisaged as aspects of the present invention. 

[0282] In view of the foregoing description it will be evident to a person skilled in the art that various modifications 
may be made within the scope of the invention. 

[0283] The scope of the present disclosure includes any novel feature or combination of features disclosed therein 
either explicitly or implicitly or any generalisation thereof irrespective of whether or not it relates to the claimed invention 
or mitigates any or all of the problems addressed by the present invention. The applicant hereby gives notice that new 
claims may be formulated to such features during tho prosecution of this application or of any such further application 
derived therefrom. In particular, with reference to the appended claims, features from dependent claims may be com- 
bined with those of the independent claims and features from respective independent claims may be combined in any 
appropriate manner and not merely in the specific combinations enumerated in the claims. 

Claims 

1 . An integrated circuit for a digital still camera, comprising: 

(a) a first programmable processor programmed to run control functions, said first process or coupled to a user 
interface, a controller for memory, and a controller for image acquisition; 

(b) a second programmable processor prog rammed to run image processing and compression functions, said 
second processor coupled to said first processor; and 
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(a) a Third processor coupled to said second processor, said third processor including at least four parallel 
multiply and accumulate units. 

A method of resizing a digital image, comprising the steps of: 

(a) providing an M1 by M2 pixel image where M1 and M2 are positive integers; 

(b) filtering a row of M1 pixels of the image with a set of filter coefficients which depends upon pixel location 
to yield M1 filtered outputs; 

(c) discarding M1-N1 of the filtered outputs of step (b); 

(d) repeating steps (b) and (c) for all other rows -of the M1 by M2 image to yield a N1 by M2 image: 

(e) filtering a column of M2 pixels of the N1 by M2 image with a set of filter coefficients which depends upon 
pixel location to yield M2 firtered outputs; 

(f) discarding M2-N2 of the filtered outputs of step (e); and 

(g) repeating steps (e) and (f) for all other rows of the N1 by M2 image to yield a N1 by N2 image. 
A method of color-filtered array interpolation, comprising the steps of: 

interpolate a first color subarray to yield a. first color array; 

high-pass filler the first color array from the prior step Lo yield a high pass first color array; 
interpolate a second color subarray io yield a second color array; 

weight and add the high-pass first color array to the second color array to yield a modified second color array; 
interpolate a third color subarray to yield a third color array; 

weight and add the high-pass first color array to the third color array to yield a modified third color array: 
defining a three-color image with the first color array, the modified second color array, and the modified third 
color array. 

A method of tone-scaling an image, comprising the steps of: 

(a) providing an M by N pixel image where M and N are positive integers and each pixel has an intensity in 
the range from 0 to K; 

(b) determining a scaled cumulative distribution function T(X) for X in the range 0 to K and with values in the 
range 0 to K. T(X) equals K multiplied by the number of pixefs with intensity less than or equal to X and then 
divided by MN; 

(c) selecting a parameter A in the range 0 to t ; 

(d) adjusting an intensity Z of a pixel of step (a) to AT(Z)+(1 -A)Z. 

A method of interpolation for a Bayer color-filtered array, comprising the steps of: 
interpolate the green subarray to form a tentative green array; 

clamp the interpolated pixel values of the tentative green array to lie in the range of the middle two values of 
the four neighboring values of the green subarray to form a green array; and 
interpolate the red and blue subarrays. 

A method of interpolation for a Bayer color-filtered array image, comprising the steps of: 

interpolate the green subarray to form a green array; 
diagonally interpolate the red subarray to form a second red subarray: 
horizontally and vertically interpolate the second red subarray to form a red array; 
diagonally interpolate the blue subarray to form a second blue subarray; 
horizontally and vertically interpolate the second blue subarray to form a blue array; 

wherein the diagonal interpolation includes weightings by the values of the corresponding pixels in the green 
array: 

wherein the horizontal and vertical interpolation includes weightings by the values of the corresponding pixels 
in the green array; and 

the green array, red array, and blue array define a color image. 
A method of white balancing for a color image sensor comprising the steps of: 
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(a) providing output green, red, and blue pixel intensities for a color image sensor illuminated with white light: 

(b) determining the ratio of the green pixel intensities to the blue pixel intensities for the outputs of step (a): 

(c) providing an adjustment for blue pixel intensities as multiplication by the ratio of step (b) plus addition of 
an offset which depends upon the pixel intensities in step (a): 

(d) determining the ratio of the green pixel intensities to the red pixel intensities for the outputs of step (a); and 
(c) providing an adjustment for red pixel intensities as multiplication by the ratio of step (b) plus addition of an 
offset which depends upon the pixel intensities in step (a). 

A method of video decoding and display, comprising the steps of: 

(a) providing a sequence of encoded video frames, a sequence of video frame boundary times ; and storage 
locations for decoded ones of the frames; 

(b) when the mth frame of the sequence has been decoded, next decode the mth frame of the sequence with 
m* determined as TP n divided by DeltaT where DeitaT is the time interval between successive ones of the 
video frame boundary times of step (a) and TP n is a presentation time determined as the greater of (i) the 
presentation time TP n _., of the mth frame plus DeltaT and (ii) the starting time Ts for decoding the frame to be 
presented next after the mth frame plus an estimated time to decode a frame, the sum rounded up to a multiple 
of DeltaT wherein the starting time Ts is the greater ot (i) current time and (ii) the time a storage focation to 
contained the decoded m'Lh frame will be available due to a prior decoded frame being displayed; 

(c) displaying a decoded frame; 

(d) repeating steps (b) and (c) with incremented indices. 

A method of interpolation for a compfementary-color-filtered array image, comprising the steps of: 

(a) provide a complementary-color-filtered array of pixel values with yellow pixel values Ye on a first subarray, 
cyan pixel values Cy on a second subarray, magenta pixel values Mg on a third subarray, and green pixel 
values G on a fourth subarray; 

(b) interpolating the yellow pixel values to all pixels in the array; 

(c) interpolating the cyan pixel values to all pixels in the 

(d) interpolating the magenta pixel values to all pixels in the array; 

(e) interpolating the green pixel values to all pixels in the array: 

(f) adjusting the pixel yellow value by subtracting quantity (Ye + Cy - 2*G - Mg) /4 from the pixel yellow value 
from step (b) where Ye is the pixel yellow value from step (b), Cy is the pixel cyan value from step (c). Mg is 
the pixel magenta value from step (d), and G is the pixel green value from step (e): 

(g) adjusting the pixel cyan value by subtracting quantity (Ye + Cy - 2*G - Mg)/4 from the pixel cyan value from 
step (c) where Ye is the pixel yellow value from step (b), Cy is the pixel cyan value from step (c), Mg is the 
pixel magenta value from step (d), and G is the pixel green value from step (e): 

(h) adjusting the pixel magenta value by adding quantity (Ye + Cy - 2*G - Mg)/4 to the pixel magenta value 
from step (d) where Ye is the pixel yellow value from srep (b) ; Cy is the pixel cyan value from step (c), Mg is 
the pixel magenta value from step (d), and G is the pixel green value from step (e); and 

(i) adjusting the pixel green value by adding quantity (Ye + Cy - 2~G - Mg)/8 to the pixel green value from step 
(e) where Ye is the pixel yellow value from step (b). Cy is the pixel cyan value from step (c), Mg is the pixel 
magenta value from step (d), and G is the pixel green value from step (e). 

A method of interpolation for a color-filtered array image, comprising the steps of: 

(a) provide a color-nitered array of pixel values with red pixel value R(j,k) for the pixel located at the jth row 
and kth column when j and k are even integers, blue pixel value B(j,k) for pixel located at jth row and klh 
column when j and k are odd integers, and green pixel value G{j ; k) forthe pixel located at the jth row and kth 
column when one of j and k is an even integer and the other an odd integer; 

(b) horizontally interpolate rows of pixel values when j is even to yield rows of red values R(j ; k) and green 
values G(j ; k) for all k : and when j is odd to yield green values G(j f k) and blue values B(j.k) for all k: 

(c) when j is an even integer vertically interpolate the horizontal interpolated red rows from step (b) to yield 
the red values R(j-1 s k) and when j is an odd integer vertically interpolate the horizontal interpolated blue to 
yield the blue values B(j-1 ,k); 

(d) vertically filter the horizontal interpolated green pixel rows from step (b) to yield filtered green output values 
G-fj.k); 

(e) define the difference values D(j,k) between the green values GQ,k)and the filtered green values G"(j.k): 



49 



EP 1 111 904 A2 



(I) when j is even subtract DQ.k) from the red values R(j,k) to yield red output values R"(j M) and when m is 
odd add D(j.k) to the red values RQM) to yield red output values R"(j t k): and 

(g) when m is even add D(j,k) to the blue values B(j.k) to yield blue output values B"(j.k) and when m is odd 
subtract D(j.k) from the blue values B(j.k) to yield blue output values B" (j .k). 

11 . A method of playback buffering, comprising: 

(a) providing first and second buffers, each of the buffers with a full/not-full indicator, and a position indicator 
indicating a position within the first and second buffers: 

(b) filling the first buffer with encoded bits when the full/not-full indicator for the first buffer indicates not-full 
and the position indicator indicates a position in the second buffer; 

(c) filling the second buffer with encoded bits when the full/not-full indicator for the second buffer indicates not- 
full and the position indicator indicates a position in the first buffer; 

(d) decoding a group of bits in the first and second buffers starting at the position indicated by the position 
indicator and treating the first and second buffers as forming a single circular buffer; 

(e) updating the fuil/not-full indicators according to the bits removed by step (d); and 

(f) repeating steps (b)-(e). 

2. A computer program comprising computer- or machinereadable computer program elements for configuring a 
computer to implement the method of any one of claims 2 to 11 . 

3. A computer program comprising computer- or machinereadable computer program elements translatable for con- 
figuring a computer to implement the method of any one of claims 2 to 11 . 

4. A carrier medium carrying a computer program according to claim 1 2 or 1 3. 



50 



EP 1 111 904 A2 



150 

CCD 
MODULE 



100- 



1G2 



PREVIEW 
ENGINE 



■104 



108 



CCD J BURST MODE 
CONTROLLER "j [COMPRESSION 



120 



OSP 



I 



122 



128 



110 



SDRAM 
CONTROLLER 



160 

_Z_ 



C> SDRAM 



IMAGE BUFFERS 



126 



IMogirg 






Extension 




VLC 


(iMX) 








ARM 



MEMORY 

r 



132 



IE 



DSP SUBSYSTEM 



FIG. 1a 



NTSC/PAL 
ENCODER 



=Otv 



106 



USB 
CORE 



I/O 
BLOCK 



■142 

loo 

140 



CCO 
MODULE 








PREVIEW ENGINE 


— i 



BURST MODE 
COMPRESSION 



SDRAM 



DSP SUBSYSTEM 





NTSC 
ENCC 


/PAL 
)DER 




IMoging 






Extension 




VLC 


(iMX) 







FIG. 2 



51 



EP1 111 904 A2 



o 
to 



to 



TT 



A 



£5 




CD 
CJ 


TERN 




UJ 



5** 



UJ 



T 
i 



m 
O 



CM 



-o 



to » 



o 

o 



ZE 



o : 



Si 



O ^ 
3 LO UU 

Log 

ac °- o 
coo i 



CM 



X 



CD 



8g 



USB 
ciever 






















>c 




RAM 


< 






O 






s 


a: 


_ J 




i 


o 


i 


Q£ 

UJ 






<. 


CD £^ 
4/1 S < 














on 












Q 

w 



















o 



5 



z o 

— u> 



CM 



2^ 



Of 

Id 



o 

-f — 



5£ HO- 



Q. 



Bl 



I 



HQ 



I 



o 



CNI 



CO 



0= 



o 



12. 



00 , 



3i 



5 >< 







r> 




CD 


























CD 






Q 




m 


o 


1 
1 




» 
1 





or 

O U- 



V 

CN 

oo 



Lu O 

oc cc 

UJ O 



52 



EP 1 111 904 A2 



ceo 

MOOUCE' 











r 


PREVIEW ENGINE ^ 







CONTROLLER 



DSP SUBSYSTEM 




SDRAM 
CONTROLLER 







iMaging 






Extension 




VLC 


(iMX) 








FIG. 3 a 



47.8 cyc/pxl= , 
1 2230 cyc/l6x16 pxl- 



SORAMC —A 



DSP 



iMX 



VLC 



\ 



EE 



3 



A— A 



\ 



9-8 



0 



— A 



A — A 



B — B 



l i MX USES BUFFER A 



0 



I 



VLC, SDRAMC, J VLC, SORAMC, 
DSP USE BUFFER B OSP USE BUFFER A 



FIG. 3b 



I B— 



A— A ^ 
iMX USES BUFFER 6 I 



B— 3 



53 



EP 1 111 904 A2 



PREVIEW 
ENGINE 



CCD 
MODULE 



c±0 



IE 



C> r- 



cco 

CONTROLLER 



BURST 
=j MOOE 
ENGINE 



05P SUBSYSTEM 



DSP 



SDRAM 
CONTROLLER 







SDRAM 





NTSC/PAL 
ENCODER 



OMA 



i 

IMAGE 
SUFFERS 





0 


IMcging 






Extension 




VLC 


(iMX) 







ARM 



MEMORY 



IL-^ I/O 
' ^ ni nn 



USB 
CORE 



BLOCK 



TV 



CFC 



FIG. 4 



CCD 
MODULE 








PREVIEW ENGINE 


i 



=j BURST MOOE 
COMPRESSION 



DSP SUBSYSTEM 




._. J. 1 

I IMAGE 4 
BUFFERS 



% 


T 


Waging 
Extension 
(iMX) 




VLC 



FIG. 5 



54 



EP1 111 904 A2 



CCD 
MODULE 



PREVIEW ENGINE 



CCD 
CONTROLLER 



_l BURST MODE 
COMPRESSION 



DSP SUBSYSTEM 



DSP 




SDRAM 
CONTROLLER 



f IMAGE BUFFERS'?" 



I 



ARM 



MEMORY 



I Waging 
Extension (iMX) 



VLC 



it. I/O 



US9 
CORE 



BLOCK 



FIG. 6 



R 


G 


R 


G 


G 


8 


G 


B 


R 


G 


R 


G 


G 


B 


G 


B 



FIG. 7a 



Ye 


cy 


Ye 


Cy 


G 


Mg 


G 


Mq 


Ye 


Cy 


Ye 


cy 


G 


Mg 


G 


Mq 



FIG. 7 b 



lf( IN[17]=1 or IN[16]=1) Ihen 

OUT[9..0]=0x3FF; 
else 

OUT(9..01=IN[15..6]; 
end if: 



dolo[9..0]- 



PVCAIN[7..0]- 



9oinl[7„0]- 
qoin2[7„0]- 
goin3[7..0]- 
goin4[7..0]- 
goiruselector.sw 



lf( IN[17J=1 or |N[16]=1 or IN[15j=l) then 
OUT[7..O]=0xFF; 

0UTt7..0]=IN[l4..7]; 
end if: 



vl8 8IT5, 


CLIP 

S 


10 BITS, ^ 








0x000' 
white_bolonce_goLn[7.0] 

FIG. 8 



dato_oui 
[7-01 



55 



EP1 111 904 A2 



dato[7 
gain[7.0] 



..o]-*{X} 

nl J 



data[l5..0] 



FIG. 9 a 



CLIP 

S 
IK 



■doia_out[7..0] 



lf( dato[15]=1 or dota(H]=1 ) then 

dota_aut=OxFF; 
else 

dota_out=doto(l3..6]; 
end if: 



OUTPUT 



FIG. 




OFFSETS 
0FFSFJ2 

OFFSET 1 



AREA1 AREA2 AREA3 255 



INPUT 



doto_in[7..0]- 

areol[7..0]- 
orea2[7..0] 
oreo3[7..0]-» 



16 BITS 



x 1/15 



AREA 
DETECTOR 



gainl[7..0]- 
gain2[7„0]- 
go;r.3[7..0]- 
goin4[7..0]- 

0x00- 
olfset2[7..0]- 
offset3[7„0]- 
offset4[7..oj- 





f 



dota[l2..0] 



CLIP 

S 



^[7..0] 




Hntn n„rf7 nl- i 255 (dolo[l 2:.0]*255) 



FIG. 9 c 



56 



EP1 111 904 A2 



IX 



O CO 



CO 
GO 



CO 



CQ 
OG 



oo 
o 



*— i 



cc p 



CD 



s- 



q 



comp2RGB 




k i 










3 


>> 






or 


\ 
O 




On 






COLOR 






ADJUSTMENT 




< 


\ 


>s 


> 


f 






CD 








COLOR 






SELECTION 








i j 










5 


O 















Is 



CM 

"o 



o 
o 



2 / 



HORIZONTAL 
DOWN SAMPLING 



in 
m 
oo 



s 



z 

uu 
a. 



o 
lo 



HORIZONTAL 
INTERPOLATION 



E5 
aO 



HORIZONTAL 
NOISE FILTER 



o 



CN 
-O 

o 



5 



LO 
» — 

CX3 
QO 



CQ 



i 



1 co 

D GO 



I 



I 



to 

h — 

as 
ao 

4 



LO 
CD 
cn 



65 



CO 

in 1 



I 



q 



57 



EP 1 111 904 A2 



RAW DATA 



G 20 



b 20 



321 



B21 



G 22 



>22 



923 



B 23 





















G oo 








G 02 








G 04 














a 03 






































G I1 








G 13 




















































G 20 








C22 














B 21 








a 23 






HORIZONTAL fl 
INTERPOLATION V 




























901 




G 02 




903 






b 00 




801 




b 02 




B 03 




b 04 


R 10 




MO 




R 12 




r 13 




R 14 


910 




G n 




912 




G U 




914 





















G 24 



b 24 



f/C. 10c 



NORMAL MOPE 

-Ri 0 +2G n +2R, 2 +2G ir R H 
912= 4 

SIMPLE MOPE 
912= 



G 11 +G 13 



2 



data 
[7-0]' 



8 BITS 



10 BITS 



■ z_1 1 


r z_1 ~~ 




r z_1 " 




0x00— j 

8 BITS 
8 BITS f 


S — H_1NTP 
V 8 BITS 





8 BITS 



x 2+1 


11 BITS 


x 1/4 





lf( !N[9]=n then 
0UT[7..0]=0x00; 
else 

tf( IN[8]=1) then 
0UT(7..0]=0xFF; 
else 

0UT[7..0]=lN[7..0]; 
end if; 
end if; 

1£ 



10 BITS 



HJNTP 



CLIP 



doto_ou( 
^ [7-01 



9 BITS 



BITS 



FIG. 10d 



58 



EP 1 111 904 A2 



OUTPUT Of HORIZONTAL INTERPOLATION 



921 



B 



21 



922 



3 22 



VERTICAL fl 
INTERPOLATION V 






r 01 


r 02 


901 
B 01 


G 02 
b 02 




r 11 


R 12 
912 


t> 1 1 


b 12 



r 21 



921 
8 21 



r 22 



922 
B 22 



COLOR ADJUSTMENT f] 
(NORMAL MODE} V 



— b Q2~ G 02 +b 22~ G 22 
b 12 = 2 912 



FIG. 10e 



f "oi 

901 
B 01 



r 11 
en 

ill 

92'. 
B21 



r 02 
G 02 
b 02 

R 12 
912 

-ELL 

r ~22 
922 
B 22 









901 


G 02 


903 


8 01 


b 02 


BOJ 




r 1l 


R 12 


r 13 


G 11 


912 


G 13 









923 



B 23 



r 03 



903 
BQ3 



r l3 
G 13 



b 13 



r 23 



923 
B 23 



f 03 
903 
B 03 

_Gl3_ 

5L 
~W 

923 
B 23 



SIMPLE MODE 



r 01 
901 



B 01 

in 
in 

921 
0 21 



*02 
G 02 
b 02 

R 12 
912 
b 12 

R 22 

922 
B 22 



f 03 
903 
B 03 

in 
in 

923 
B 23 



59 



EP 1 111 904 A2 



OUTPUT OF HORIZONTAL INTERPOLATION 



Ye 00 


cyoo 






G 10 


m 9Q0 


Y <20 


^20 





y«oi 


c yoi 





311 "911 



X e 21 



VERTICAL f) 
INTERPOLATION V 



Ye 00 


cyoo 


900 


m goo 


y*1Q 


c m 


G 10 


m 9l0 


Ye 20 


^20 


920 


mg 2 o 



5*01 


c yoi 


901 


mgoi 


y e n 




911 


Mg M 


y e 2i 


c y2i 


321 


mg 21 



COLOR ADJUSTMENT fl 
(NORMAL MOOE) V 



y*oo 




900 


m 900 


y e io 


cyio 


910 


m gio 


> e 20 


^20 


920 


mg 20 



Y «02 


C V02 





'12 



mg 12 



Ye 22 icy 22 



Ye 0 2 


^02 


902 


mg 02 


y e i2 


^12 


G 12 


mg 12 


Ye 22 cy 22 


922 


™922 



y^ol 


cyoi 


y e 02 


^02 


901 


"^Ol 


902 


mg Q2 


y e n 


c yii 


y«i2 


^12 


9TT 




I 912 


mg 12 


y e 2i 


<*2i 


y*22 




921 


mg 2 i 


922 


m 922 








SIMPLE MODE 



y e oo 


cyoo 


900 


"^00 


y e io 


c yio 


G 10 


m 9l0 


Ye 20 


^20 


920 


mg 2 o 



y e oi 


c yoi 


901 




y e n 


cyii 


911 


M 911 


y«2i 


Cy2i 


921 


m 921 



a=g n +Mg |r ye n -cy u 

y«TT=y«ii+f 
cyTT =c yn + i 

9n = 9irx 
1[ ™9TT™ M 9ll — §- 



Y '02 


^02 


902 


mg Q2 


^12 


^12 


Gi 2 




Ye 22 c/22 


922 


m 922 



FIG. 10 f 



60 



EP1 111 904 A2 



HORIZONTAL INTERPOLATION OUTPUTS 



Goo 



b 00 



Rio 



910 



c 20 



b 20 



901 



'01 



11 



'11 



921 



021 



G 02 



b 02 



( 12 



912 



G 22 



b 22 



903 



B 03 



13 



G 13 



923 



hi 



G Q4 



b 04 



<14 



9H 



Q 2 4 



>24 



VERTICAL 
INTERPOLATION 



•L121 
RJ21 
L101 

•R_101 



NOISE FILTER = OFF 



L_121=R 12 
R_l21=g 12 



NOISE FILTER = ON 

^ G 0 2-H2g l2 ^22 
L_121=R,2-91 2 + 4 



R_121 = 



L101 = 



G Q2'*" 2 9l2' l ' G 22 
4 

G 02+ G 22 
2 

b 02 +b 22 



JW. 10g 



61 



EP 1 111 904 A2 




62 



EP 1 111 904 A2 





U21 




VERTICAL 
INTERPOLATION 


RJ21 j 


COLOR 
SELECTION 


L_>01 


R_1C1 











NOISE FILTER = OFF 

R/YesR^ 

G/Mg=g 1 ^ 
9/Cy=^22 

B/Cy=^22 



■C/Mg 
■8/Cy 
■9/G 



NOISE FILTER = ON 
R/V.-B n , G 02 +2 912 +C 22 

g ^ = Gq2+29i2+G22 



g/Cy= 



G02+G22 



B/Cy= b02^22 



FIG. 10i 



lefLIOI- 
riqhLIOI- 

lefL12l- 
righl_121- 




cfcLpottcrn- 



COLOR EXTRACTOR 




cfo_ pattern — »• COLOR EXTRACTOR 



-o 
-o 



cfo.pattern— » COLOR EXTRACTOR 



RAe 



-B/Cy 



G/Mg 




FIG. 1 Oj da_paitern— *■ COLOR EXTRACTOR 



63 



EP1 111 904 A2 




a 
o 

O 
<^> 



7 ? M J] M 



X 



on 
55 





CN 







r— ^ CO 
>5> a* cx> 



On 




CM 









3E 



CO 
CO 
CN 



On 



kO 



CD 

II 
o 



CD 



64 



EP 1 111 904 A2 




65 



EP 1 111 904 A2 



R[7..0]- 
G[7..0]- 

B[7..0]. 



coef1[7. 



coef2[7..0] 



H8H 

.0]-J 

lJ 



coef3 



coef4| 



coefSl 



coef6f 



coef7f 



coefSl 



coef9| 



— <8HI 

xoH 

'7..0H 

H8HI 

7..0H 
7..0]-J 
7..0]-J 

7..0]-f 
7.0] -J 



/64 



/64 



(Yln<0) 

Y[7..0]=^255 (Yin>255) 
Yln[7..0] (others) 



Yin[l1..0] 



CLIP 

y 



-Y[7..0] 



/64 



Z«tKB 



/64 



(CbinS-123) 
Cbt7..0]=-{ 255 (Cbini127) 
Cbin[7..0]+128 (others) 

12 BnS^/T\Cbin[12..(l] 



CLIP 



-Ct)[7..0] 



offseLCb[7..0] 



/64 



/64 



/64 



(CrinS-128) 
Cr[7..0]=^255 (Crin*127) 
Crin[7..0]+128 (others) 



12 BITS /V\Crin(l2..0] 



CLIP 
y-f-~Cr[7..0j 



offseLCr(7..0] 



TO. 7* / a 



OUTPUT . 
ENHANCEMENT SIGNAL 



FIG. 1 1 b 







4 

v SLOPE Si 


12- 


X. 








INPUT 


/ 




EDGE SIGNAL 



66 



EP 1 111 904 A2 



FIG. 12a 



BURST MODE 
COMPRESS AND 
DECOMPRESS 



DSP 



PREVIEW 
ENCINE 



CCD DATA 
BUFFER 



SDRAM 



SDRAM CONTROLLER 



IMAGE 
BUFFER 



DATA 
BUFFER 



ARM 



EXTERNAL 
FLASH/CFC 



OSD 



VIDEO 
ENCODER 



PERIPHERAL 
MODULES 

req [0] 
reqen [I] 
address [0] 
odata [OJ 
idata [|] 
bs [0] 
rw [0] 
dten [0] 
elk [I] 


»■ 


SDRAM 
CONTROLLER 

[I] req ros [0] 
[0] reqen cos [0] 
[l] address we [0] 
[I] odota cs [0] 
[0] idata bonk [0] 
[I] bs dqm [0] 
[I] rw address [0] 
[I] dten dola [10] 
[I] elk eke [0] 





































FIG. 12b 



TO 
SDRAM 



IN 

1 

DPCM 

~r~ 

HUFFMAN 
ENCODING 



ENCOOEO 
DATA 



t 




HUFFMAN 
TA8LE 






HUFFMAN 




OECOOINC 






l-DPCM 





FIG. 13b 



motSSLe- 



PREVIEW ENGINE 



CCDC 



COMP ENGINE 
DEC0UP ENGINE ■ ^ 



TV ENC 



SDRAM 
CTRL 



IMAGE BUFFER 

HI 

dsp ;mx vlc 



J 

ARM 



CJC> SDRAM 



FIG. 13a 



67 



EP1 111 904 A2 



CO 



!= 

CD 
QO 



3 

CO 







SOr 






r ' 


4:2:2 


-¥ 


'4 


CD 
Q£ 


YCbCr 



o 

2^ 



l — 
GQ 
00 



<: 
o 



CD 



o 



T 



s 



en 
» — 

CD 



1 — 
CD 



CD 



CFA 








WHITE 


BALANCE 



CD 

.<=> 



2s 




68 



EP 1 111 904 A2 




XO.YO 






1 






XI, Y1 



FIG. 1 6 



FIG. 17 



FIG. 18a 



1 22 







DSP EXT 


DSP 




MEM t/F 



A 

512x64b 



iMX 



9 

51 2x64b 



SD 

1 



■124 



VLC 



■125 



iMX COEFF 




iMX GMD 




VLC QMAT 




VLC HUFF 


1280x48b 




512x16b 




512x15b 




1024x32b 



OOOOh 

0080h 

7F80h 
aOOOh 
COOOh 



FFBOh 
FFFFh 



(MP/MC.=0) 



COVLY=T) 



32k OA 
(0VLY=1) 



OOOOh 
0060h 
0080h 

7FB0h 
SOOOh 
COOOh 



FFFFh 



MUR 



32k OA 
(0VLY=1) 



FIG. 18b 



69 



EP 1 111 904 A2 



QJnp 



md_add 




FIG. 19 



>> 



• md_add 



CMP 



\ 



CMP 



saLhiqh 



7 



+ ■* — lookup^base 



j 



laokup_oddr 

FIG. 21 



SHARED COEFF 
MEMORY MEMORY 



CATA 
MEMORY 
WRITE 



ARBITER 



COEFF 
MEMORY 
READ 



OATA 
MEMORY 
READ 



OWR 



COMMAND 
MEMORY 



AOORESS GEM 



ORO 



ORD 



INPUT FORMATTER 



OATA PATH 



DRO 



COMMAND 
DECODE 



EXECUTION 
CONTROL 



FIG. 20 



70 



EP 1 111 904 A2 



DSP EXT 
MEM l/F' 



REG l/F 



SHARED DATA 
MEMORY A/B ' 



ARBITER 



CONTROL AND REGISTERS 



—►•ENCODE PIPELINE 



Ja ' D 



QMAT 



0 



HufLtbl 



DECOOE PIPELINE-*- i 



FIG. 22 



32K WORD RAM 




HPIB 


DSP 





EM l/F 



BUS CONTROLLER 



ARM7TDMI 



EXTERNAL MEMORY 



FIG. 23a 



0{l5:O] 



cs_hpib 



cs_dspc 



o 
o 
s 

CO 



HCNTLO 



HCNTL1 



HDt7:0] 



5409 SUBCHIP 



HPI BLOCK 



HPl BLOCK 
REGISTER 
SELECT 



HPIA[15:0] 




ADDRESS, 


32K 
WORD 
DARAM 




DATA 


HPIC[15:0] 


HPI D[ 15:0] 







FIG. 23b 



71 



EP 1 111 904 A2 



0000:0000h 

0QO0:00Q3h 
0000:0004h 

0000:7fFFh 
O003:0000h 
0003:0FFFh 

001 0:01 OOh 
0010;FFFFh 
0200:0000h 
027F:FFFFh 
0400:0000h 
047F:FFFFh 
0800:0000h 
OFFF:FFFFh 



ARM PROGRAM/ 
DATA SPACE 



RESET VECTOR 
(BRANCH CODE) 
4 BYTES 



INTERNAL RAM 
~ 32 KBYTES 



RESERVED 



INTERNAL 
PERIPHERALS 
+ KBYTES 



RESERVED 



DSP MEMORY 
64 KBYTES 



RESERVED 



EXTERNAL FLASH 
8 MBYTES 



RESERVED 



CFC 
8 MBYTES 



RESERVED 



SORAM 
128 MBYTES 



OOOOh 
O060h 
008Oh 



7F80h 
8000H 

COOOh 



FFFFh 



DSP DATA SPACE 



MMR 



SCRATCH -PAD RAM 



ON-CHIP 
32K WORD DARAM 
(OVLY=i) 



RESERVED 



IMAGE 8UFFER 



RESERVED 



FIG. 23c 



TDl _ 
TOO 

ARM7 TDM I 



FIG. 24 



TDl 



TOO 



C54X 



CO-EMUUVTiON LOGIC 



TDl 

•nTRST 
TCK 
TMS 
TOO 



■ EMU 1 
•EMU 0 



72 



EP 1 111 904 A2 




















NO 


8° 





CM 

o 

CO 



5; 



si 



o a 



s 

5 



2: 

8 



5 



> 



I 

o 



5 J 






3: 




O 


Q 


c_> 


UJ 


1— 




UJ 




o 





a: *— 


o cn 




— I o 


O O 







a ill 



CD 



C3 



TONE 
SCALING 



COLOR 
CORRECTION 



CD O 



WHITE 
BALANCE 







i 




CD 


O 


Of 





CFA 

INTERPOLATION 
1 

35 S 



73 



EP 1 111 904 A2 




FIG. 28 




m 



V77A 

WA4 



fat 



7777* 

m 



w 



177771 



PI 



Kc3 



m 



Pi 



||| ALREADY HAS CREEN 



C ! DOES NOT HAVE GREEN 
G IS BEING INTERPOLATED 



8-2 



5x5 FILTER 



CUP 



INTERPOLATED 
VALUE 



G1 








G2 








G3 








G4 





o 

i 

O 



I I 

B-1 



f/C. 29 



DISCARD THE SIGGE5T 
AND THE SMALLEST 



74 



EP 1 111 904 A2 



O ^3 

u a J 

oe o gd 



o 
uu a: 



a: 
O 













>- 






>- 








CK 


o 






O 


3 

LU 






s 


2 








UJ 
Z 






5 




-i 









.J 



So 
£i a. 
cc a: 
O y 



gig 

* <2 >- 



CO 



^3 

5 



CN* 
4- 

a: 



cn 
-t- 
c 



cm e 

to 



c c 

— <"v_^ 

ti ii 

CN 



ILTER 
0.5 




LTER 
0.5 






U. ~- 






1 IT) 
(XL "j 


St 

§§ 




3TAPS Fl 



c 



o 



o 



c 

o 



on 



Q O 



o o 



CN 
-»- 

c 



i 

o 



I 

J, 



I 

C5 



c 

i 

CN 

I 

*5f 



CN CN 

-+- -+- 

c c 

— " CSJ 

t ] 

£ CD 

IT + 
O 



i 



CN 



£ CD 0_ 

t5* "C* 

C - I I LU 

£ p p cr 
>S ^ 

O CD O X 



O - 

°- h 

— z 

< £n 

<: u 

to <^ 

— o 

x a: 

O 1 



o 



a_ 



o 

2 

LU 
3 



o 



CD 



>- 
O 



>- 



c 
cr 



CN 
C 

E" 
o* 



'cT 



CO 

5 



75 



EP1 111 904 A2 



.C(m-I.x),..- 
,.R(m-2.x),. 

..G(m,x),., 
.B(m-l,x)... 



G{m-1.x) 



NO USE 



•8(m-1.x) 



.G(m-2,x),.. — -NO USE 



FIG. 33 a 



.G(m+l,x),.. — -NO USE 
..R(m,x),.. 



■ R(m,x) 



.C(m-1,x),.. — -NO USE 



.3(m+1.x),..- 
..G(m,x),..- 
-B(m-1.x),..- 



1/2 



■ B(m,x)=(B(m- 1 ,x)+E(m+ 1 .x))/2 
G(m,x) 



FIG. 33b 



RAW DATA 
(CFA IMAGE)" 



o 

CL 

cx 

o 
o 



FIG. 34 



UNE MEMORY 



LIKE MEMORY 



LINE UEMORY 



LINE MEMORY 



UNE MEMORY 



LINE MEMORY 



LINE MEMORY 



LINE MEMORY 



or 

UJ 

o 

LU 
CO 
O 

5 



5 

O 

Q_ 



■RED 

•GREEN 

•BLUE 



76 



EP 1 111 904 A2 




77 



EP 1 111 904 A2 




78 



EP 1 1 1 1 904 A2 



..G(m,x) 
.G(m-1.x) 
.G(m-2.x) 




..B(m-I.x)..- 
..R(m,x)..- 
..R(m-2.x)..- 



■*G"(m-U) 



Delto_G(m-1,x) 



^ -8"(m-1,> 



1/2 



-R"(m-1.x) 



FIG. 37a 



..G(m+1.x).< 
..G(m,x).. 
..G(m-I.x). 

„R(m,x)„ - 
..B(m-M).." 
..B(m+1,x)..- 




Delto_C(m,x) 



9 



1/2 



FIG. 37b 



■R"(m.R) 
-B"{m. x ) 




TO. 38 



79 



EP 1 111 904 A2 



1 

0.8 

OUTPUT o s - 
VOLTAGE 

0.4 h 
0.2 
20 




- FIG. 39a 



40 50 60 
INTENSITY 





80 



EP1 111 904 A2 



0 12 3 4 5 

I II! — i — h 



i t i 



61 62 63 64 65 



61 




62 



-1 0 1 

FIG. 41b 



6,3 



FIG. 41a 



10 



10 



xxxxxxxxxxxx 



xxxxxxxxxxixx 

N K K K K K „ K. 



FIG, 42a 




FIG. 4Zb 



8 



xxxxxxxxl 



xxxxxxxxl 



xxxxxx"xxl 



xxxxxxxxl 



xxxxxxxx 



xxxxxxxxl 

FIG. 42 e 



81 



EP 1 111 904 A2 




82 



EP 1 111 904 A2 



LUMINANCE - 



B 

HISTOGRAM 
EQUALIZATION 



■432 



CONVERTER LUMINANCE 
„ r * , _ LUMINANCE - 

FIG. 43 





HISTOGRAM 
EQUALIZATION 


^-442 


v 444 




a EFFECT 


C 




-440 




CONVERTER 


LUMINANCE 



FIG. 44 




0 5Q0 1000 1500 2000 2500 3000 3500 4000 

FIG. ' 45 



Tdl 



Td2 I 



Td3 j| ♦ • «~m[ Tdm j 



DECODING TIME 



FRAMEq 


FRAME, 


FRAME 2 


FRAME3 


• ■ » 


FRAME m 


» • » 



PRESENTATION 
TIME 



AT 



2AT 



3AT 4AT 

FIG. 46a 



• » • 



mAT 



TdO 



Tdl ] l [ Td3 j) [•»•; ) [ 1 Tdm] [ j ) [ OECODING TIME 



FRAMEq FRAME 



FRAME 1 



FRAME3 



FRAME 



m 



PRESENTATION 
T1UE 



AT 2AT 3AT 4AT 

FIG. 46b 



t 



83 



EP 1 111 904 A2 



or. 
i — 
in 



o 
o 
o 




BUFFER (TP 0 ,S 0 ) 



BUFFER n-1 (TP^.S^) 

BUFFER n (TP n ,S„) 
BUFFER n+1 (TP„ +1 .S n+1 ) 



BUFFER N-1 (TP^.^Sn.,) 

FIG. 47 




SLICE 0 
SLICE 1 
SLICE 2 



SLICE n 



MACROBLOCK 



FIG. 49 



SDRAM 
CONTROLLER 



=0 



31STREAM 
BUFFER 



SDRAM 



3E 



VIDEO 
DECODER 



F/G. 50 



OISPLAY 



F/C. 51a 



Ps 



Ps 



BUFFER STATUS 
AFTER DECODING A UNIT 



Pd 



RE-FILLED BUFFER BEFORE 
DECODING THE NEXT UNIT 



Pd 




BUFFER STATUS 
AFTER DECODING A UNIT 



^ RE -FILLED BUFFER BEFORE 
DECODING THE NEXT UNIT 

FIG. 51b 



64 



EP1 111 904 A2 



( START ) 



S n =TP n =0, 0*n£N 
n=t=m'=fim=0 



S n =1, SKIP Am FRAMES 
DECOOE FRAME m" 



Ts=t, n++ 
COMPUTE TP n , m\ Am 




FIG. 48 



n=N f , t=0. T=0 




END OF \_ YES 
SEQUENCE 



Ts=mox|T, TP n+1 j 
COMPUTE TP n . m\ Am 




S n =0, SKIP Am FRAMES 
DECODE FRAME m" 




T=l - Td+=DT* m=fn * 



S„=1. T=t. n++ 



85 



EP1 111 904 A2 



PS 



LEFT BUFER 



RIGHT BUFFER 







I 

r v 



BUFFER STATUS AFTER INITIALIZATION 



STATUS 0 



I 




Pa 




BUFFER STATUS AFTER DECODING A UNIT 



BUFFER STATUS AFTER RE-FILLING BUFFER 



Ps 



STATUS 1 ) 



I 



NOT-FULL 








\ J 








\ 





SUFFER STATUS AFTER OECOOING A UNIT 



I 



fttm, 


NOT-FULL 




) 1 



BUFFER STATUS AFTER DECODING A UNIT 



Ps 



STATUS 3 



FIG. 52 



86 



(19) 



3 



Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 



(12) 



(88) Date of publication A3: 

16.03.2005 Bulletin 2005/11 

(43) Date of publication A2: 

27.06.2001 Bulletin 2001/26 

(21) Application number: 00311430.3 

(22) Date of filing: 20.12.2000 



(H) EP 1 111 904 A3 

EUROPEAN PATENT APPLICATION 

(51) IntCI. 7 : H04N 1/40 



(84^ Desinnatpri Contra fit inn States* 


• Yam sn if* h i Qotrtrii 
TdlTlaUCni, OdLOlU 


AT BE CH CY DE DK ES Fl FR GB GR IE IT LI LU 




MC NL PT SE TR 


• Zhou, Minhua 


Hfisinnatpd f-xtpn^ion Statp^' 


Planrt Tpyac fl IQ^ 


AL LT LV MK RO SI 


• llgner, Klaus 




81549 Miinchen (DE) 


(30) Priority: 20.12.1999 US 172780 P 


• Talluri, Rajendra 


29.06.2000 US 215000 P 


Piano, 75203 (US) 


29.06.2000 US 214951 P 


• Yoo, Youngjun 


04.08.2000 US 632543 P 


Richardson, Texas 75080 (US) 


14.01.2000 US 176272 P 


• Liang, Jie 




Piano, Texas 75025 (US) 


(71 ) Applicant: Texas Instruments Incorporated 


• Tsai, Mandy 


Dallas, Texas 75251 (US) 


Taipei 111 (KR) 




• Tsunoda, Kiyoshi 


(72) Inventors: 


Tokohama-city, Kanagawa (JP) 


• Hung, Chlng-Yu 


• Inamorl, Shinri 


Piano, Texas 75024 (US) 


Yokohama-shi, Kanagawa (JP) 


• Tamama, Hideo 




Tsukuba, Ibaraki (JP) 


(74) Representative: Potter, Julian Mark et al 


• Osamoto, Akira 


D Young & Co 


Inashiki, Ibaraki (JP) 


120 Holborn 


• Koshiba, Osamu 


London EC1N 2DY (GB) 


Tsukuba, Ibaraki (JP) 





CO 

< 

O 
O) 



CL 
LU 



(54) Digital camera system and method 

(57) Digital Still Camera (DSC) includes separate 
preview engine, burst mode compression/decompres- 
sion engine, image pipeline, CCD plus CCD controller, 
and memory plus memory controller. ARM microproc- 
essor and DSP share control. Color filter array interpo- 
lation by use of green high-frequency added to red and 
blue plus interpolation. 




100- 



120^1 





] k 126 


Dlogin? 
QQtnaan 

(art) 


VIC 



BP SUBSYSTEV 



USB 
CORE 



I/O 



-142 

££> 
MO 



FIG. 1a 



Primed by Jouvc. 75001 PARIS (FR) 



BMSDOCfO <EP. 



.UM904A3_I_> 



EP1 111 904 A3 



European Patent 
Office 



EUROPEAN SEARCH REPORT 



Application Number 

00 31 1430 



DOCUMENTS CONSIDERED TO BE RELEVANT 



Category 



Citation of document with indication, where appropriate, 
of relevant passages 



Relevant 
to claim 



CLASSIFICATION OF THE 
APPLICATION (lnt,CI.7) 



EP 0 675 648 A (EASTMAN KODAK CO) 
4 October 1995 (1995-10-04) 
* abstract; figures 1,2 * 



HO4N1/40 



Th e presen t search report hna been dro v w up for otl oloima 



TECHNICAL FIELDS 
SEARCHED (lnt.CI.7) 



H04N 



Place or seaicn 



The Hague 



Dale ol completion ol the search 

5 October 2004 



E«aminer 

Isa, S 



CATEGORY OF CITED OOCLIME NTS 

X : particularly relevant il taken alone 

Y : partfcu/a/ly tetuvani if combined with another 

document o! the same category 
A : technological backg round 
O : non-writlen disclosure 
P : intermediate document 



T : theory or principle underlying the invention 
E : earlier patent document, but published on. or 

after the filing date 
D . docuiiivfti citud in the applicator? 
L : docurnem cited tor other reasons 

& : member of the same patent tamity. coi responding 
document 



2 



DNSOOCIO <EP. 



11 11904A3_I_> 



EP1 111 904 A3 



European Patent 
Office 



EP GO 31 1430 



Application Number 



CLAIMS INCURRING FEES 



The present European patent application comprised at the time of filing more than ten claims. 

□ Only part of the claims have been paid within the prescribed time limit. The present European search 
report has been drawn up for the first ten claims and for those claims for which claims fees have 
been paid, namely claim(s): 



□ No claims fees have been paid within the prescribed time limit. The present European search report has 
been drawn up for the first ten claims. 



I The Search Division considers that the present European patent application does not comply with the 
requirements of unity of invention and relates to several inventions or groups of inventions, namely: 



see sheet B 



□ All further search fees have been paid within the fixed time limit. The present European search report has 
been drawn up for all claims. 



□ As ail searchable claims could be searched without effort justifying an additional fee. the Search Division 
did not invite payment of any additional fee. 



Only part of the further search fees have been paid within the fixed time limit. The present European 
search report has been drawn up for Ihose parts of the European patent application which relate lo the 
inventions in respect of which search fees have been paid, namely claims: 



None Of the further search fees nave been paid within the fixed time limit. The present European search 
report has been drawn up for those parts of the European patent application which relate to the invention 
first mentioned in the claims, namely claims: 



LACK OF UNITY OF INVENTION 




1 



3 



BNSDOCID: <EP. 



U 1 1904A3_J_> 



EP 1 111 904 A3 



Application Number 

EP GO 31 1430 

The Search Division considers lhal the present European patent application does not comply with the 
requirements of unity of invention and relates to several inventions or groups of tnventions ; namely: 

1. claim: 1 

Intergrated circuit for a digital still camera comprising 
three programable processors 

2. claims: 2, and claims 12-14 as far as dependent on claim 2 

A method for resizing a digital image 

3. claims: 3, and claims 12-14 as far as dependent on claim 3 

A method of colour-filtered array interpolation 

4. claims: 4, and claims 12-14 as far as dependent on claim 4 

A method of tone-scaling an image 

5. claims: 5, and claims 12-14 as far as dependent on claim 5 

A method of interpolation for a Bayer colour-filtered array 

6. claims: 6, and claims 12-14 as far as dependent on claim 6 

A method of interpolation for a Bayer colour-filtered array 
image 

7. claims: 7, and claims 12-14 as far as dependent on claim 7 

A method of white balancing for a colour image sensor 

8. claims: 8, and claims 12-14 as far as dependent on claim 8 

A method of video decoding and display 

9. claims: 9, and claims 12-14 as far as dependent on claim 9 

A method of interpolation for a 
complementary-colour-filtered array image 

10. claims: 10, and claims 12-14 as far as dependent on claim 10 

A method of interpolation for a colour-filtered array image 



3 



European Patent LACK OF UNITY OF INVENTION 
0ff,Ce SHEET B 



4 



BNSDOCiD: <EP. 



1 111904A3_L> 



EP 1 111 904 A3 




European Patent LAC|< QF y^-py Qf , NVENT|0N App.ication Nun,t>er 

6 SHEET B EP OQ 31 1430 



The Search Division considers that the present European patent application does not comply with the 
requirements of unity of invenlion and relates to several inventions or groups of inventions, namely: 



11. claims: 11, and claims 12-14 as far as dependent on claim 11 
A method of playback buffering 



BNSDOCIO <EP 1 1 1 1 904A3J_> 



5 



EP 1 111 904 A3 



ANNEX TO THE EUROPEAN SEARCH REPORT 

ON EUROPEAN PATENT APPLICATION NO. EP 00 31 1430 



This annex lists the patent family members relating to the patent documents cited in the above-mentioned European search report. 
The members are as contained in the European Patent Office EDP file on 

The European Patent Office is in no way liable for these particulars which are merely given for the purpose of information. 

05-10-2004 



Patent document 


Publication 


Patent famiiy 
members) 


Publication 


cited in search report 


date 


date 


EP 0675648 A 


04-10-1995 


us 


5477264 A 


19-12-1995 






DE 


69522597 Dl 


18-10-2001 






DE 


69522597 T2 


11-07-2002 






EP 


1152589 A2 


07-11-2001 






EP 


1152590 A2 


07-11-2001 






EP 


0675648 A2 


04-10-1995 






JP 


3597249 B2 


02-12-2004 






JP 


7274060 A 


20-10-1995 






JP 


2004048802 A 


12-02-2004 



For more details about this annex : see Olficial Journal of the European Patent Office, No. 12/82 



6 



1U1904A3_I_> 



J) 



Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 



(11) 



EP 1 111 904 A8 



CORRECTED EUROPEAN PATENT APPLICATION 

Note: Bibliography reflects the latest situation 



Correction information: 
Corrected version no 1 
INIDcode(s) 30 



(W1 A2) 



(51) lntC!7: H04N 1/40 



Corrigendum issued on: 
16.08.2001 Bulletin 2001/33 



(19) 



(12) 



(15) 



(48) 



(43) Date of publication: 

27.06.2001 Bulletin 2001/26 

(21) Application number: 00311430.3 

(22) Date of filing: 20.12.2000 



(84) Designated Contracting States: 

AT BE CH CY DE DK ES R FR GB GR IE IT LI LU 
MCNLPTSETR 
Designated Extension States: 
AL LT LV MK RO SI 

(30) Priority: 20.12.1999 US 172780 P 
29.06.2000 US 215000 P 
29.06.2000 US 214951 P 
04.08.2000 US 632543 P 
14.01.2000 US 176272 P 

(71) Applicant: Texas Instruments Incorporated 
Dallas, Texas 75251 (US) 

(72) Inventors: 

• Hung, Ching-Yu 
Piano, Texas 75024 (US) 

• Tama ma, Hideo 
Tsukuba, Ibaraki (JP) 

• Osamoto, Akira 
Inashiki, Ibaraki (JP) 

• Koshiba, Osamu 
Tsukuba, Ibaraki (JP) 



(54) Digital camera system and method 



• Yamauchi, Satoru 
Tsuchiura, Ibaraki (JP) 

• Zhou, Minhua 

Piano, Texas 75093 (US) 

• llgner, Klaus 
81549 Munchen (DE) 

• Talluri, Rajendra 
Piano, 75203 (US) 

• Yoo, Youngjun 
Richardson, Texas 75080 (US) 

• Liang, Jie 

Piano, Texas 75025 (US) 

• Tsai, Mandy 
Taipei 111 (KR) 

• Tsunoda, Kiyoshi 
Tokohama-city, Kanagawa (JP) 

• Inamori, Shinri 
Yokohama-shi, Kanagawa (JP) 

(74) Representative: Potter, Julian Mark et al 
D. Young & Co., 
21 New Fetter Lane 
London EC4A 1 DA (GB) 



CO 

< 

o 



Q. 

LU 



(57) Digital Still Camera (DSC) includes separate 
preview engine, burst mode compression/decompres- 
sion engine, image pipeline, CCD plus CCD controller, 
and memory plus memory controller- ARM microproc- 
essor and DSP share control. Color filter array interpo- 
lation by use of green high-frequency added to red and 
blue plus interpolation. 




Printed by Jouve. 75001 PARIS (FR) 



BNSDOCID: <EP. 



11U904ABJ_> 



